七天入门LLM大模型 | 第五天：大模型自动评估理论和实战–LLM Automatic Evaluation

# Examples in BBH
Evaluate the result of a random Boolean expression.
Q: not ( ( not not True ) ) isA: Let's think step by step.Remember that (i) expressions inside brackets are always evaluated first and that (ii) the order of operations from highest priority to lowest priority is "not", "and", "or", respectively.We first simplify this expression "Z" as follows: "Z = not ( ( not not True ) ) = not ( ( A ) )" where "A = not not True".Let's evaluate A: A = not not True = not (not True) = not False = True.Plugging in A, we get: Z = not ( ( A ) ) = not ( ( True ) ) = not True = False. So the answer is False.
Q: True and False and not True and True isA: Let's think step by step.Remember that (i) expressions inside brackets are always evaluated first and that (ii) the order of operations from highest priority to lowest priority is "not", "and", "or", respectively.We first simplify this expression "Z" as follows: "Z = True and False and not True and True = A and B" where "A = True and False" and "B = not True and True".Let's evaluate A: A = True and False = False.Let's evaluate B: B = not True and True = not (True and True) = not (True) = False.Plugging in A and B, we get: Z = A and B = False and False = False. So the answer is False.

# Demo -- model_genereate 直接生成response
def model_generate(query: str, infer_cfg: dict) -> str:
    inputs = tokenizer.encode(query)    input_ids = inputs['input_ids']    ...
    # Process infer cfg (do_sample, top_k, top_p, temperature, special_tokens ...)    generation_config = process_cfg(args)    ...      # Run inference    output_ids = model.generate(        input_ids=input_ids,        attention_mask=attention_mask,        generation_config=generation_config,    )
    response = tokenizer.decode(output_ids, **decode_kwargs)    return response

# Demo -- model_call方式计算loglikelihood
# context + continuation 拼接，示例：# Question：法国的首都是哪里？# Choices：A.北京  B.巴黎  C.汉堡  D.纽约  # pair-1: (ctx, cont) = (法国的首都是哪里？，A.北京)  # pair-2: (ctx, cont) = (法国的首都是哪里？，B.巴黎)  # pair-3: (ctx, cont) = (法国的首都是哪里？，C.汉堡)  # pair-4: (ctx, cont) = (法国的首都是哪里？，D.纽约)# Logits --> 
def loglikelihood(self, inputs: list, infer_cfg: dict = None) -> list:    # To predict one doc    doc_ele_pred = []    for ctx, continuation in inputs:
        # ctx_enc shape: [context_tok_len]  cont_enc shape: [continuation_tok_len]        ctx_enc, cont_enc = self._encode_pair(ctx, continuation)
        inputs_tokens = torch.tensor(            (ctx_enc.tolist() + cont_enc.tolist())[-(self.max_length + 1):][:-1],            dtype=torch.long,            device=self.model.device).unsqueeze(0)
        logits = self.model(inputs_tokens)[0]        logits = torch.nn.functional.log_softmax(logits.float(), dim=-1)
        logits = logits[:, -len(cont_enc):, :]        cont_enc = cont_enc.unsqueeze(0).unsqueeze(-1)        logits = torch.gather(logits.cpu(), 2, cont_enc.cpu()).squeeze(-1)
        choice_score = float(logits.sum())        doc_ele_pred.append(choice_score)
    # e.g. [-2.3, 1.1, -12.9, -9.2], length=len(choices)    return doc_ele_pred

# 1. 代码下载git clone [email protected]:modelscope/llmuses.git
# 2. 安装依赖cd llmuses/pip install -r requirements/requirements.txtpip install -e .

# 指定模型和数据集python llmuses/run.py --model ZhipuAI/chatglm3-6b --datasets ceval --outputs ./outputs/test --limit 10

python llmuses/run.py --model ZhipuAI/chatglm3-6b --outputs ./outputs/test2 --model-args revision=v1.0.2,precision=torch.float16,device_map=auto --datasets arc --limit 10

# Examplepython llmuses/run_arena.py --c registry/config/cfg_single.yaml --dry-run

# Examplepython llmuses/run_arena.py --dry-run --c registry/config/cfg_pairwise_baseline.yaml

python llmuses/run_arena.py -c registry/config/cfg_arena.yaml --dry-run

# Usage:streamlit run viz.py -- --review-file llmuses/registry/data/qa_browser/battle.jsonl --category-file llmuses/registry/data/qa_browser/category_mapping.yaml