Skip to content

Commit 855ded1

Browse files
authored
Merge pull request #183 from VectorInstitute/b/fix-doc
Fix doc
2 parents feed010 + f556960 commit 855ded1

2 files changed

Lines changed: 6 additions & 2 deletions

File tree

docs/index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ pip install vec-inf
1616
Otherwise, we recommend using the provided [`vllm.Dockerfile`](https://github.com/VectorInstitute/vector-inference/blob/main/vllm.Dockerfile) and [`sglang.Dockerfile`](https://github.com/VectorInstitute/vector-inference/blob/main/sglang.Dockerfile) to set up your own environment with the package. The built images are available through [Docker Hub](https://hub.docker.com/orgs/vectorinstitute/repositories)
1717

1818
If you'd like to use `vec-inf` on your own Slurm cluster, you would need to update the configuration files, there are 3 ways to do it:
19+
1920
* Clone the repository and update the `environment.yaml` and the `models.yaml` file in [`vec_inf/config`](https://github.com/VectorInstitute/vector-inference/blob/main/vec_inf/config), then install from source by running `pip install .`.
2021
* The package would try to look for cached configuration files in your environment before using the default configuration. The default cached configuration directory path points to `/model-weights/vec-inf-shared`, you would need to create an `environment.yaml` and a `models.yaml` following the format of these files in [`vec_inf/config`](https://github.com/VectorInstitute/vector-inference/blob/main/vec_inf/config).
2122
* [OPTIONAL] The package would also look for an enviroment variable `VEC_INF_CONFIG_DIR`. You can put your `environment.yaml` and `models.yaml` in a directory of your choice and set the enviroment variable `VEC_INF_CONFIG_DIR` to point to that location.

docs/user_guide.md

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ You should see an output like the following:
3737
```
3838

3939
**NOTE**: You can set the required fields in the environment configuration (`environment.yaml`), it's a mapping between required arguments and their corresponding environment variables. On the Vector **Killarney** Cluster environment, the required fields are:
40+
4041
* `--account`, `-A`: The Slurm account, this argument can be set to default by setting environment variable `VEC_INF_ACCOUNT`.
4142
* `--work-dir`, `-D`: A working directory other than your home directory, this argument can be set to default by seeting environment variable `VEC_INF_WORK_DIR`.
4243

@@ -63,12 +64,14 @@ vec-inf launch Meta-Llama-3.1-8B-Instruct --vllm-args '--max-model-len=65536,--c
6364
```
6465

6566
For the full list of inference engine arguments, you can find them here:
66-
* [vLLM](https://docs.vllm.ai/en/stable/serving/engine_args.html).
67-
* [SGLang](https://docs.sglang.io/advanced_features/server_arguments.html)
67+
68+
* [vLLM: `vllm serve` Arguments](https://docs.vllm.ai/en/stable/serving/engine_args.html)
69+
* [SGLang: Server Arguments](https://docs.sglang.io/advanced_features/server_arguments.html)
6870

6971
#### Custom models
7072

7173
You can also launch your own custom model as long as the model architecture is supported by the underlying inference engine, and make sure to follow the instructions below:
74+
7275
* Your model weights directory naming convention should follow `$MODEL_FAMILY-$MODEL_VARIANT` ($MODEL_VARIANT is OPTIONAL).
7376
* Your model weights directory should contain HuggingFace format weights.
7477
* You should specify your model configuration by:

0 commit comments

Comments
 (0)