Improve training setup and continued pretraining docs

[Rohithmatham12]

Summary

• update environment pins that caused setup and streaming dataset failures: fastparquet, snappy, and datasets
• document Linux/NVIDIA setup assumptions, channel-priority guidance, and FlashAttention installation for Llama 2 32K training
• add a continued-pretraining script template for Llama-2-7B-32K-beta with environment-variable overrides
• document dataset licensing separately from repository code licensing

Addresses #162.
Addresses #169.
Addresses #174.
Addresses #175.
Addresses #184.
Also helps the related setup discussion in #119.

Testing

• bash -n training/continued_pretrain_llama-2-7b-32k.sh
• bash -n training/finetune_llama-2-7b-32k-mqa.sh training/finetune_llama-2-7b-32k-booksum.sh
• python3 -c "import pathlib; text=pathlib.Path('environment.yml').read_text(); assert 'datasets==2.15.0' in text; assert 'snappy=1.1.10' in text; assert 'fastparquet=2023.7.0' in text; print('environment pin check ok')"
• git diff --check

[broly-code-security-scanner]

Broly Security Scan

Note

Baseline snapshot is missing for this repo. Broly is running in PR-only fallback mode until the first scheduled baseline completes. This does not block the PR.

Note

✅ Clean scan
No vulnerabilities detected in this PR.

Note

Re-scan this PR anytime with /broly scan — useful after /broly undismiss, or to refresh findings without a new push.

Broly — SAST (zai-org/GLM-5.2) · Secrets · SCA · GH Actions (zizmor) · Containers · SBOM · Powered by Together AI