Intro to Running LLMs Locally

This guide covers the what, how, and why of running LLMs locally using llama.clj, a clojure wrapper for the llama.cpp library.

Large language models (LLMs) are tools that are quickly growing in popularity. Typically, they are used via an API or service. However, many models are available to download and run locally even with modest hardware.

The One Basic Operation

From the perspective of using an LLM, there's really only one basic operation:

That's basically it. All other usage derives from this one basic operation.

Recreating the Chat Interface

If you've interacted with an LLM, it's probably while using one of the various chat interfaces. Before exploring other usages of local LLMs, we'll first explain how a chat interface can be implemented.

Tokens

Keen readers may have already noticed that chat interfaces work with text, but LLMs work with tokens. Choosing how to bridge the gap between text and tokens is an interesting topic for creating LLMs, but it's not important for understanding how to run LLMs locally. All we need to know is that text can be tokenized into tokens and vice versa.

Just to get a sense of the differences between tokens and text, let's look at how the llama2 7b chat model tokenizes text.

One thing to notice is that there are fewer tokens than letters:

If we untokenize each token, we can see that tokens are often whole words, but not always.

Just to get a feel for a typical tokenizer, we'll look at some basic stats.

Number of tokens:

The longest token:

Token with the most spaces:

One last caveat to watch out for when converting between tokens and text is that not every individual token produces a valid utf-8 string. It may require multiple tokens before a valid utf-8 string is available.

Fortunately, llama.clj has a utility for untokenizing that will take care of the issue:

Prediction

Returning to the one basic operation, we now know how to translate between text and tokens. Let's now turn to how prediction works.

While our description of the one basic operation says that LLMs calculates probabilities, that's not completely accurate. Instead, LLMs calculate logits which are slightly different. Even though logits aren't actually probabilities, we can mostly ignore the details except to say that larger logits indicate higher probability and smaller logits indicate lower probability.

Let's take a look at the logits for the prompt "Clojure is a".

clojure-is-a-logits is an array of numbers. The number of logits is 32,000 which is the number of tokens our model can represent. Each index in the array is proportional to the probability that the corresponding token will come next according to our LLM.

Given that higher numbers are more probable, let's see what the top 10 candidates are:

And for comparison, let's look at the 10 least probable candidates:

As you can see, the model does a pretty good job of finding likely and unlikely continuations.

Full Response Generation

Generating probabilities for the very next token is interesting, but not very useful by itself. What we really want is a full response. The way we do that is by using the probabilities to pick the next token, then append that token to our initial prompt, then retrieve new logits from our model, then rinse and repeat.

One of the decisions that most LLM APIs hide is the method for choosing the next token. In principle, we can choose any token and keep going (just as we were able to choose the initial prompt). The name for choosing the next token using the logits provided by the LLM is called sampling.

Choosing a sampling method is an interesting topic unto itself, but for now, we'll go with the most obvious method. We'll choose the token with the highest likelihood given by the model. Sampling using the highest likelihood option is called greedy sampling. Conventionally, greedy sampling isn't the best sampling method, but it's easy to understand and works well enough.

Ok, so we now have a plan for generating a full response:

1. Feed our initial prompt into our model
2. Sample the next token using greedy sampling
3. Return to step #1 with the sampled token appended to our previous prompt

But wait! How do we know when to stop? LLMs define a token that llama.cpp calls end of sentence or eos for short (end of stream would be a more appropriate name, but oh well). We can repeat steps #1-3 until the eos token is the most likely token.

Finally, one last note before we generate a response is that chat models typically have a prompt format. The prompt format is a bit arbitrary and different models will have different prompt formats. Since the prompt format is defined by the model, users of models should check the documentation for the model being used.

Since, we're using llama2's 7b chat model, the prompt format is as follows:

Let's see how llama2 describes Clojure.

See llama2's response below. Note that the response includes the initial prompt since the way we generate responses simply appends new tokens to the initial prompt. However, most utilities in llama.clj strip the initial prompt since we're usually only interested in the answer generated by the LLM.

Let's ask a follow up question. All we need to do is keep appending prompts and continue generating more tokens.

We've now implemented a simple chat interface using the one basic operation that LLMs offer! To recap, LLMs work by calculating the likelihood of all tokens given a prompt. Our basic process for implementing the chat interface was:

1. Feed our prompt into the LLM using the prompt structure specified by our chosen LLM.
2. Sample the next token greedily and feed it back into the LLM.
3. Repeated the process until we reached the end of sentence (eos) token.

Reasons for Running LLMs Locally

Now that we have a general sense of how LLMs work, we'll explore other ways to use LLMs and reasons for running LLMs locally rather than using LLMs through an API.

Privacy

One reason to run LLMs locally rather than via an API is making sure that sensitive or personal data isn't bouncing around the internet unnecessarily. Data privacy is important for both individual use as well as protecting data on behalf of users and customers.

Alternative Sampling Methods

Sampling is the method used for choosing the next token given the logits returned from an LLM. Our chat interface example used greedy sampling, but choosing the next token by always selecting the highest likelihood token often does not lead to the best results. The intuition for greedy sampling's poor performance is that always picking the highest probability tokens often leads to boring, uninteresting, and repetitive results.

Let's compare greedy sampling vs mirostatv2, llama.clj's default sampling method:

mirostatv2 response:

greedy response:

Evaluating the outputs of LLMs is a bit of a dark art which makes picking a sampling method difficult. Regardless, choosing or implementing the right sampling method can make a big difference in the quality of the result.

To get a feel for how different sampling methods might impact results, check out the visualization tool at https://perplexity.vercel.app/.

Constrained Sampling Methods

In addition to choosing sampling methods that improve responses, it's also possible to implement sampling methods that constrain the responses in interesting ways. Remember, it's completely up to the implementation as to determine which token gets fed back into the model.

Run On Sentences

It's possible to arbitrarily select tokens. As an example, let's pretend we want our LLM to generate run-on sentences. We can artificially choose "and" tokens more often.

By artificially boosting the chances of selecting "and", we were able to generate a rambling response. It's also possible to get rambling responses by changing the prompt to ask for a rambling response. In some cases, it's more effective to artificially augment the probabilities offered by the LLM.

This is a pretty naive strategy and improvements are left as an exercise to the reader. As a suggestion, two easy improvements might be to use a better model or pay more attention to the probabilities rather than having sharp cut offs (ie. boosting at every five tokens and only considering the top 30 results).

JSON Output

We can also use more complicated methods to constrain outputs. For example, we can force our response to only choose tokens that satisfy a particular grammar.

In this example, we'll only choose tokens that produce valid JSON.

Note: This example uses a subset of JSON that avoids sequences that would require lookback to validate. Implementing lookback to support arbitrary JSON output is left as an exercise for the reader.

Classifiers

Another interesting use case for local LLMs is for quickly building simple classifiers. LLMs inherently keep statistics relating various concepts. For this example, we'll create a simple sentiment classifier that describes a sentence as either "Happy" or "Sad". We'll also run our classifier against the llama2 uncensored model to show how model choice impacts the results for certain tasks.

Our implementation prompts the LLM to describe a sentence as either happy or sad using the following prompt:

We then compare the probability that the LLM predicts the response should be "Happy" vs the probablility that the LLM predicts the response should be "Sad".

In this example, the llama2 uncensored model vastly outperforms the llama2 model. It was very difficult to even find an example where llama2 would label a sentence as "Sad" due to its training. However, the llama2 uncensored model had no problem classifying sentences as happy or sad.

More Models Options

New models with different strengths, weaknesses, capabilities, and resource requirements are becoming available regularly. As the classifier example showed, different models can perform drastically different depending on the task.

Just to give an idea, here's a short list of other models to try:

• metharme-7b: This is an experiment to try and get a model that is usable for conversation, roleplaying and storywriting, but which can be guided using natural language like other instruct models.
• GPT4All: GPT4All is an ecosystem to train and deploy powerful and customized large language models that run locally on consumer grade CPUs.
• OpenLLamMa: we are releasing our public preview of OpenLLaMA, a permissively licensed open source reproduction of Meta AI’s LLaMA
• ALMA: ALMA (Advanced Language Model-based trAnslator) is an LLM-based translation model, which adopts a new translation model paradigm: it begins with fine-tuning on monolingual data and is further optimized using high-quality parallel data. This two-step fine-tuning process ensures strong translation performance.
• LlaMa-2 Coder: LlaMa-2 7b fine-tuned on the CodeAlpaca 20k instructions dataset by using the method QLoRA with PEFT library.

Conclusion

LLMs really only have one basic operation which makes them easy to learn and easy to use. Having direct access to LLMs provides flexibility in cost, capability, and usage.

Next Steps

For more information on getting started, check out the guide.