Large files
Where large files and data go: release assets, a Zenodo record, a Hugging Face dataset, then Git LFS for small binaries only.
GitHub blocks a file larger than 100 MiB in git, and warns above 50 MiB (the limits). Data rarely belongs in git at all: it has its own homes, which cost you nothing and give it a persistent identifier.
Where data goes, in this order
- Release assets. Attach the files to a release of your repository on GitHub: each fileunder 2 GiB, up to 1,000 files a release, with no limit on their total size or on downloads. Source: About releases.
- A Zenodo record, which you make on Zenodo: a DOI for the data, cited by the paper and by the repository's README. Zenodo can also archive each release of a GitHub repository (Zenodo and GitHub).
- A Hugging Face dataset in your own Hugging Face account, for data that is read in parts or updated (datasets,storage limits).
- Git LFS, for small binaries the code needs next to it: a figure's source file, a trained model of modest size, a stimulus file. Its quotas are per account, so one popular dataset in LFS could spend a month's downloads for all your repositories.
OSCR never downloads your LFS objects or release assets itself: the downloads are counted on GitHub, as anyone's.
Git LFS support
GitHub supports Git LFS. Git keeps a small pointer file (a version line, the object's SHA-256 and its size) in place of each large file, and the file itself goes to GitHub's LFS storage. Which files are LFS files is written in the repository's .gitattributes. Sources: About Git LFS,Git LFS.
Client setup
Install Git LFS once on each computer (installing Git LFS), then in the repository track a pattern and commit .gitattributes:
git lfs install git lfs track "*.nii.gz" git add .gitattributes git commit -m "Track NIfTI images with Git LFS"
Everyone who clones the repository needs Git LFS too: without it, they get the pointer files instead of the content. Say so in the README. Sources: Configuring Git LFS,Collaboration with Git LFS.
LFS file size and quotas
On GitHub Free, one LFS file is at most 2 GB; each account has 10 GiB of LFS storage and 10 GiB of LFS downloads a month, for all its repositories. A fork's LFS objects count against the parent repository's owner, and pushing a file again counts it in full. Sources:About Git LFS, Git LFS billing.
Over quota
Once a quota is spent on an account without a payment method, Git LFS is blocked for that account until the next month: clones get the pointer files only, and new LFS pushes are refused. With a payment method on file, GitHub bills the excess instead. OSCR asks for no payment method and pays for nothing on your behalf. Source: Git LFS billing.
Integrity check
GitHub refuses a push whose commits point to LFS objects that were never uploaded. Upload every object the branches point to, then push again:
git lfs push --all origin
Source: Resolving Git LFS upload failures.
File locking
A binary file cannot be merged, so two people editing it at once lose one person's work. Git LFS can lock it: a pattern marked lockable is read-only in each clone until you lock the file; GitHub refuses a push that changes a file someone else has locked; an administrator can force an unlock.
git lfs track "*.fig" --lockable git lfs lock figures/figure-a.fig git lfs locks git lfs unlock figures/figure-a.fig git lfs unlock figures/figure-a.fig --force
Source: Git LFS's locking.
Moving files into and out of LFS
Into LFS, for a file already committed: git lfs migrate rewrites the history so that every commit holds the pointer. Rewriting history has consequences for tracing maps: readrewriting history first.
git lfs migrate import --include="*.h5" --everything git push --force-with-lease --all origin
Out of LFS, from now on:
git lfs untrack "*.h5" git rm --cached data/results.h5 git add data/results.h5 .gitattributes git commit -m "Keep results.h5 in git, not LFS"
Objects removed from LFS still count against the storage quota: GitHub deletes them only with the repository, or through GitHub Support. Sources: Moving a file to Git LFS,Removing files from Git LFS.
LFS objects in archives
GitHub's "Download ZIP" and release archives hold the LFS pointer files unless the repository's setting "Include Git LFS objects in archives" is on. When it is, the archives hold the files, and each download counts against the LFS download quota. Source: Managing Git LFS objects in archives.
The binary and line-ending attributes
.gitattributes also tells git how to treat text and binary files: text=auto lets git normalise line endings, eol=lf or eol=crlf fixes them for a pattern, andbinary (the same as -text -diff) tells git never to change a file nor show it line by line.
# .gitattributes * text=auto *.sh text eol=lf *.bat text eol=crlf *.mat binary *.nii.gz binary
Sources: Configuring Git to handle line endings,gitattributes.