Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 3 additions & 13 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,28 +1,18 @@
PYTHON ?= python3
ZENSICAL ?= zensical

.PHONY: update-branches
update-branches:
@echo "Updating GitHub branch links in docs..."
@$(PYTHON) scripts/update_doc_links.py

.PHONY: prepare
prepare:
@$(PYTHON) scripts/prepare_docs.py

.PHONY: build
build: prepare
build:
@$(ZENSICAL) build --clean

.PHONY: serve
serve: prepare
serve:
@$(ZENSICAL) serve

.PHONY: help
help:
@echo "Makefile targets:"
@echo " prepare - Stage local docs plus imported upstream tool docs into .generated/docs"
@echo " build - Build the Zensical site with the active Python environment"
@echo " serve - Serve the Zensical site with the active Python environment"
@echo " update-branches - Update docs to point to configured development branches for repos"
@echo " help - Show this message"

Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ We use Gen3's role based access control (RBAC) to manage access to data.
* /programs/ucl
* /programs/manchester

Designated users within each institution have privileges to update requests. "Update" in this context means setting the status of a user's request to [SIGNED].
Designated users within each institution have privileges to update requests. "Update" in this context means setting the status of a user's request to `SIGNED`.

Since this approach relies on Gen3's [Requestor](https://github.com/uc-cdis/requestor/blob/master/docs/functionality_and_flow.md#example-backend-flow) for all assignments of policies to users we get the following benefits:

Expand Down
82 changes: 57 additions & 25 deletions docs/tools/funnel/docs/compute/gcp-batch.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,37 @@ menu:
weight: 20
---

!!! warning "Current Limitations ⚠️"
- Latest Funnel release ([v0.11.11](https://github.com/ohsu-comp-bio/funnel/releases/tag/v0.11.11)) requires specific bucket prefixing in the inputs and outputs
- e.g. `/mnt/disks/<BUCKET>/<FILE>` instead of `/<FILE>`
- Nextflow workflows are currently not supported (as Nextflow expects root-level working directories → `/work`)

# Overview

The following steps illustrate how to run a TES tasks via GCP Batch utilizing Google Storage Buckets.
The following steps illustrate how to run TES tasks via GCP Batch utilizing Google Storage Buckets.

GCS buckets are automatically mounted inside task containers. Input and output paths use standard filesystem paths (e.g. `/input/file.txt`) — no `/mnt/disks/` prefixing required. Nextflow workflows are supported via the TES executor `workdir` field.

# Quick Start

## 1. Download Funnel

```sh
curl -fsSL https://calypr.github.io/funnel/install.sh | bash
curl -fsSL https://calypr.org/funnel/install.sh | bash
```

## 2. Start Server

<details>
<summary><code>Config Example</code></summary>

funnel server run --Compute "gcp-batch" \
--GCPBatch.Project "example-project" \
--GCPBatch.Location "us-central1"
```yaml
Compute: gcp-batch

GCPBatch:
Project: example-project
Location: us-central1
```

</details>

```sh
funnel server run --Compute "gcp-batch" --GCPBatch.Project "example-project" --GCPBatch.Location "us-central1"
```

## 3. Submit Task
Expand All @@ -35,22 +49,23 @@ funnel server run --Compute "gcp-batch" \
"name": "Input/Output Test",
"inputs": [
{
"url": "gs://tes-batch-integration/README.md",
"path": "/mnt/disks/tes-batch-integration/README.md"
"url": "gs://my-bucket/input/README.md",
"path": "/input/README.md"
}
],
"outputs": [
{
"url": "gs://tes-batch-integration/README.md.sha256",
"path": "/mnt/disks/tes-batch-integration/README.md.sha256"
"url": "gs://my-bucket/output/README.md.sha256",
"path": "/output/README.md.sha256"
}
],
"executors": [
{
"image": "alpine",
"command": [
"sha256sum",
"/mnt/disks/tes-batch-integration/README.md | tee /mnt/disks/tes-batch-integration/README.md.sha256"
"sh",
"-c",
"sha256sum /input/README.md | tee /output/README.md.sha256"
]
}
]
Expand All @@ -75,24 +90,25 @@ funnel task get <TASK ID>
"executors": [
{
"command": [
"sha256sum",
"/mnt/disks/tes-batch-integration/README.md | tee /mnt/disks/tes-batch-integration/README.md.sha256"
"sh",
"-c",
"sha256sum /input/README.md | tee /output/README.md.sha256"
],
"image": "alpine"
}
],
"id": "d6f0tgpurbu7o23pgj20",
"inputs": [
{
"path": "/mnt/disks/tes-batch-integration/README.md",
"url": "gs://tes-batch-integration/README.md"
"path": "/input/README.md",
"url": "gs://my-bucket/input/README.md"
}
],
"name": "GCP Batch Task Example",
"name": "Input/Output Test",
"outputs": [
{
"path": "/mnt/disks/tes-batch-integration/README.md.sha256",
"url": "gs://tes-batch-integration/README.md.sha256"
"path": "/output/README.md.sha256",
"url": "gs://my-bucket/output/README.md.sha256"
}
],
"state": "COMPLETE"
Expand All @@ -102,8 +118,24 @@ funnel task get <TASK ID>
## 5. Verify Outputs

```sh
gsutil cat gs://tes-batch-integration/README.md.sha256
9b9916cea5348edd6ad78893231edb81fc96772d1dd99fae9c2a64f84646cb1c /mnt/disks/tes-batch-integration/README.md
gsutil cat gs://my-bucket/output/README.md.sha256
9b9916cea5348edd6ad78893231edb81fc96772d1dd99fae9c2a64f84646cb1c /input/README.md
```

# Nextflow

Nextflow tasks can specify a working directory via the TES executor `workdir` field, which maps to Docker's `--workdir` flag inside the GCP Batch container:

```json
{
"executors": [
{
"image": "nextflow/nextflow:latest",
"command": ["nextflow", "run", "main.nf"],
"workdir": "/work"
}
]
}
```

# Additional Resources
Expand Down
1 change: 0 additions & 1 deletion docs/tools/funnel/docs/compute/grid-engine.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,4 +54,3 @@ The following variables are available for use in the template:

See https://golang.org/pkg/text/template for information on creating templates.

[ge]: http://gridscheduler.sourceforge.net/documentation.html
4 changes: 1 addition & 3 deletions docs/tools/funnel/docs/development/developers.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ A Funnel development environment includes:
- [gRPC Gateway][gateway] for HTTP communication.
- [Angular][angular] and [SASS][sass] for the web dashboard.
- [GNU Make][make] for development tasks.
- [Docker][docker] for executing task containers (tested with v1.12, v1.13).
- [Docker](https://docker.io) for executing task containers (tested with v1.12, v1.13).
- [dep][dep] for Go dependency vendoring.
- [Make][make] for development/build commands.
- [NodeJS][node] and [NPM][npm] for web dashboard development.
Expand Down Expand Up @@ -64,8 +64,6 @@ mock interfaces in test code, for example, to mock the Google Cloud APIs.
[grpc]: http://www.grpc.io/
[sass]: http://sass-lang.com/
[make]: https://www.gnu.org/software/make/
[docker]: https://docker.io
[python]: https://www.python.org/
[dep]: https://golang.github.io/dep/
[node]: https://nodejs.org
[npm]: https://www.npmjs.com/
Expand Down
1 change: 0 additions & 1 deletion docs/tools/funnel/docs/download.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,4 +32,3 @@ Funnel requires Go 1.21+. Check out the [development docs][dev] for more detail.


[dev]: ./development/developers.md
[docker]: https://docker.io
1 change: 0 additions & 1 deletion docs/tools/funnel/docs/metrics/prometheus.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,5 +32,4 @@ Funnel exports these metrics:
of bytes of disk space available by all nodes.

[prom]: https://prometheus.io/
[gauge]: https://prometheus.io/docs/concepts/metric_types/#gauge
[graf]: https://grafana.com/
4 changes: 2 additions & 2 deletions docs/tools/grip/docs/clients.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,13 +89,13 @@ This represents the vertex we queried for above. All vertexes in the system will
* _\_id_: This represents the global identifier for this vertex. In order to draw edges between different vertexes from different data sets we need an identifier that can be constructed from available data. Often, the `_id` will be the field that you query on as a starting point for a traversal.
* _\_label_: The label represents the type of the vertex. All vertexes with a given label will share many property keys and edge labels, and form a logical group within the system.

The data on a query result can be accessed as properties on the result object; for example `result[0].data.symbol` would return:
The data on a query result can be accessed as properties on the result object; for example `result&#91;0&#93;.data.symbol` would return:

```python
u'TP53'
```

You can also do a `has` query with a list of items using `gripql.within([...])` (other conditions exist, see the `Conditions` section below):
You can also do a `has` query with a list of items using `gripql.within(&#91;...&#93;)` (other conditions exist, see the `Conditions` section below):

```python
result = G.V().hasLabel("Gene").has(gripql.within("symbol", ["TP53", "BRCA1"])).render({"_id": "_id", "symbol":"symbol"}).execute()
Expand Down
4 changes: 1 addition & 3 deletions docs/tools/grip/docs/databases.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ Drivers:

# SQLite

GRIP supports storing vertices and edges in [SQLite]
GRIP supports storing vertices and edges in [SQLite](https://sqlite.org/)

Config:

Expand All @@ -116,5 +116,3 @@ Drivers:
Sqlite:
DBName: tester/sqliteDB
```

[psql]: https://sqlite.org/
10 changes: 5 additions & 5 deletions docs/tools/grip/docs/queries/jsonpath.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,11 +60,11 @@ Below is a table of field and the values they would reference in subsequent trav
| _id | "NM_007294.3:c.4963_4981delTGGCCTGACCCCAGAAG" |
| _label | "variant" |
| type | "deletion" |
| publications[0].pmid | 29480828 |
| publications[:].pmid | [29480828, 23666017] |
| publications.pmid | [29480828, 23666017] |
| $gene.symbol.hugo | "BRCA1" |
| $gene.transcripts[0] | "ENST00000471181.7" |
| `publications[0].pmid` | 29480828 |
| `publications[:].pmid` | `[29480828, 23666017]` |
| `publications.pmid` | `[29480828, 23666017]` |
| `$gene.symbol.hugo` | "BRCA1" |
| `$gene.transcripts[0]` | "ENST00000471181.7" |


## Usage Example:
Expand Down
2 changes: 1 addition & 1 deletion docs/tools/sifter/docs/inputs/sqldump.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Scan file produced produced from sqldump.
| Name | Type | Description |
|-------|---|--------|
| path | string | Path to the SQL dump file |
| tables | []string | Names of tables to read out |
| tables | `[]string` | Names of tables to read out |

## Example

Expand Down
2 changes: 1 addition & 1 deletion docs/tools/sifter/docs/inputs/table.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Extract data from tabular file, includiong TSV and CSV files.
|-------|---|--------|
| path | string | File to be transformed |
| rowSkip | int | Number of header rows to skip |
| columns | []string | Manually set names of columns |
| columns | `[]string` | Manually set names of columns |
| extraColumns | string | Columns beyond originally declared columns will be placed in this array |
| sep | string | Separator \\t for TSVs or , for CSVs |

Expand Down
2 changes: 1 addition & 1 deletion docs/tools/sifter/docs/transforms/clean.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Remove fields that don't appear in the desingated list.

| name | Type | Description |
| --- | --- | --- |
| fields | [] string | Fields to keep |
| fields | `[]string` | Fields to keep |
| removeEmpty | bool | Fields with empty values will also be removed |
| storeExtra | string | Field name to store removed fields |

Expand Down
2 changes: 1 addition & 1 deletion docs/tools/sifter/docs/transforms/fieldProcess.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ each of the items in the array will become an independent row.
| name | Type | Description |
| --- | --- | --- |
| field | string | Name of field to be processed |
| mapping | map[string]string | Project templated values into child element |
| mapping | `map[string]string` | Project templated values into child element |
| itemField | string | If processing an array of non-dict elements, create a dict as `{itemField:element}` |


Expand Down
2 changes: 1 addition & 1 deletion docs/tools/sifter/docs/transforms/lookup.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Using key from current row, get values from a reference source
| --- | --- | --- |
| replace | string (field path) | Field to replace |
| lookup | string (template string) | Key to use for looking up data |
| copy | map[string]string | Copy values from record that was found by lookup. The Key/Value record uses the Key as the destination field and copies the field from the retrieved records using the field named in Value |
| copy | `map[string]string` | Copy values from record that was found by lookup. The Key/Value record uses the Key as the destination field and copies the field from the retrieved records using the field named in Value |
| tsv | TSVTable | TSV translation table file |
| json | JSONTable | JSON data file |
| table | LookupTable | Inline lookup table |
Expand Down
4 changes: 2 additions & 2 deletions docs/tools/sifter/docs/transforms/project.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ Populate row with templated values

| name | Type | Description |
| --- | --- | --- |
| mapping | map[string]any | New fields to be generated from template |
| rename | map[string]string | Rename field (no template engine) |
| mapping | `map[string]any` | New fields to be generated from template |
| rename | `map[string]string` | Rename field (no template engine) |


# Example
Expand Down
2 changes: 1 addition & 1 deletion docs/tools/sifter/docs/transforms/reduce.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Using key from rows, reduce matched records into a single entry
| method | string | Method name |
| python | string | Python code string |
| gpython | string | Python code string run using (https://github.com/go-python/gpython) |
| init | map[string]any | Data to use for first reduce |
| init | `map[string]any` | Data to use for first reduce |

## Example

Expand Down
2 changes: 1 addition & 1 deletion netlify.toml
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
[build]
command = "python scripts/prepare_docs.py && zensical build --clean"
command = "zensical build --clean"
publish = "site"
1 change: 1 addition & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
zensical
termynal
mkdocs-multirepo-plugin
Loading