RAG

RAG Model

The RAG model is a retrieval-augmented generation model that can be leveraged for question-answering tasks using RagTokenForGeneration or RagSequenceForGeneration as proposed in Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela.

It was added to the library in PyTorch with the following checkpoints:

• facebook/rag-token-nq
• facebook/rag-sequence-nq
• facebook/rag-token-base
• facebook/rag-sequence-base

Contributions:

• RAG #6813 (@ola13)
• [RAG] Add attention_mask to RAG generate #7373 (@patrickvonplaten)
• [RAG] Add missing doc and attention_mask to rag #7382 (@patrickvonplaten)
• [Rag] Fix wrong usage of num_beams and bos_token_id in Rag Sequence generation #7386 (@patrickvonplaten)
• [RAG] Fix retrieval offset in RAG's HfIndex and better integration tests #7372 (@lhoestq)
• [RAG] Remove dependency on examples/seq2seq from rag #7395 (@ola13)
• [Rag] fix rag retriever save_pretrained method #7399 (@patrickvonplaten)
• [RAG] Clean Rag readme in examples #7413 (@ola13)
• [RAG] Model cards - clean cards #7420 (@patrickvonplaten)
• Document RAG again #7377 (@sgugger)

Bug fixes and improvements

• Mark big downloads slow #7325 (@sgugger)
• [Bug Fix] The actual batch_size is inconsistent with the settings. #7235 (@HuangLianzhe)
• Fixed results of SQuAD-FR evaluation #7313 (@psorianom)
• [s2s] add supported architecures to MD #7252 (@sshleifer)
• Add num workers cli arg #7322 (@chadykamar)
• [s2s] add src_lang kwarg for distributed eval #7300 (@sshleifer)
• [s2s] only save metrics.json from rank zero #7331 (@sshleifer)
• [code quality] fix confused flake8 #7309 (@stas00)
• [testing] skip decorators: docs, tests, bugs #7334 (@stas00)
• Fixed evaluation_strategy on epoch end bug #7340 (@WissamAntoun)
• Models doc #7345 (@sgugger)
• Ensure that integrations are imported before transformers or ml libs #7330 (@dsblank)
• [Benchmarks] Change all args to from no_... to their positive form #7075 (@fmcurti)
• Remove reference to args in XLA check #7344 (@ZeroCool2u)
• wip: Code to add lang tags to marian model cards #6586 (@sshleifer)
• Expand a bit the documentation doc #7350 (@sgugger)
• Check decorator order #7326 (@sgugger)
• Update modeling_tf_longformer.py #7359 (@Line290)
• Updata tokenization_auto.py #6870 (@hjptriplebee)
• Update the TF models to remove their interdependencies #7238 (@jplu)
• Make PyTorch model files independent from each other #7352 (@sgugger)
• Clean RAG docs and template docs #7348 (@sgugger)
• Fixing case in which Trainer hung while saving model in distributed training #7365 (@TevenLeScao)
• Formatter #7368 (@LysandreJik)
• [seq2seq] make it easier to run the scripts #7274 (@stas00)
• Remove mentions of RAG from the docs #7376 (@sgugger)
• [fsmt] build/test scripts #7257 (@stas00)
• [s2s] distributed eval allows num_return_sequences > 1 #7254 (@sshleifer)
• Seq2SeqTrainer #6769 (@patil-suraj)
• modeling_bart: 3 small cleanups that dont change outputs #7381 (@sshleifer)
• Check config type using type instead of isinstance #7363 (@LysandreJik)
• [s2s, examples] minor doc changes #7385 (@patil-suraj)
• Remove unhelpful bart warning #7391 (@sshleifer)
• [code quality] new make target that combines style and quality targets #7310 (@stas00)
• Speedup check_copies script #7394 (@sgugger)
• Fix BartModel output documentation #7390 (@sgugger)
• Fix FP16 and attention masks in FunnelTransformer #7374 (@sgugger)
• [Longformer, Bert, Roberta, ...] Fix multi gpu training #7272 (@patrickvonplaten)
• [s2s] add create student script #7290 (@patil-suraj)
• [s2s] rougeLSum expects \n between sentences #7410 (@sshleifer)
• [T5] allow config.decoder_layers to control decoer size #7409 (@sshleifer)
• Flos fix #7384 (@marrrcin)
• Catch PyTorch warning when saving/loading scheduler #7401 (@sgugger)
• Pull request template #7392 (@LysandreJik)
• Reorganize documentation navbar #7423 (@sgugger)