guinmoon / llmfarm_core.swift

Swift library to work with llama and other large language models.

LLMFarm_core.swift

LLMFarm_core swift library to work with large language models (LLM). It allows you to load different LLMs with certain parameters.
Based on ggml and llama.cpp by Georgi Gerganov.

Also used sources from:

• rwkv.cpp by saharNooby.
• Mia by byroneverson.

Features

• MacOS (13+)
• iOS (16+)
• Various inferences
• Various sampling methods
• Metal (dont work on intel Mac)
• Model setting templates
• LoRA adapters support (read more)
• LoRA train support
• LoRA export as model support
• Restore context state (now only chat history)

Inferences

• LLaMA
• GPTNeoX
• Replit
• GPT2 + Cerebras
• Starcoder(Santacoder)
• RWKV (20B tokenizer)
• Falcon
• MPT
• Bloom
• StableLM-3b-4e1t
• Qwen
• Yi models
• Deepseek models
• Mixtral MoE
• PLaMo-13B

Note: For Falcon, Alpaca, GPT4All, Chinese LLaMA / Alpaca and Chinese LLaMA-2 / Alpaca-2, Vigogne (French), Vicuna, Koala, OpenBuddy (Multilingual), Pygmalion/Metharme, WizardLM, Baichuan 1 & 2 + derivations, Aquila 1 & 2, Mistral AI v0.1, Refact, Persimmon 8B, MPT, Bloom select llama inferece in model settings.

Sampling methods

• Temperature (temp, tok-k, top-p)
• Tail Free Sampling (TFS)
• Locally Typical Sampling
• Mirostat
• Greedy
• Grammar (dont work for GPTNeoX, GPT-2, RWKV)
• Classifier-Free Guidance

Installation

git clone https://github.com/guinmoon/llmfarm_core.swift

Swift Package Manager

Add llmfarm_core to your project using Xcode (File > Add Packages...) or by adding it to your project's Package.swift file:

dependencies: [
  .package(url: "https://github.com/guinmoon/llmfarm_core.swift")
]

Build and Debug

To Debug llmfarm_core package, do not forget to comment .unsafeFlags(["-Ofast"]) in Package.swift. Don't forget that the debug version is slower than the release version.

To build with QKK_64 support uncomment .unsafeFlags(["-DGGML_QKK_64"]) in Package.swift.

Usage

More examples in the Demo Project

Example generate output from a prompt

import Foundation
import llmfarm_core

let maxOutputLength = 256
var total_output = 0

func mainCallback(_ str: String, _ time: Double) -> Bool {
    print("\(str)",terminator: "")
    total_output += str.count
    if(total_output>maxOutputLength){
        return true
    }
    return false
}



let ai = AI(_modelPath: "/Users/guinmoon/dev/alpaca_llama_etc/llama-2-7b-chat-q4_K_M.gguf",_chatName: "chat")
var params:ModelAndContextParams = .default
params.promptFormat = .Custom
params.custom_prompt_format = """
SYSTEM: You are a helpful, respectful and honest assistant.
USER: {prompt}
ASSISTANT:
"""
var input_text = "State the meaning of life"
params.use_metal = true

_ = try? ai.loadModel_sync(ModelInference.LLama_gguf,contextParams: params)

let output = try? ai.model.predict(input_text, mainCallback)

Projects based on this library

• LLM Farm

App to run LLaMA and other large language models locally on iOS and MacOS.