1 Commits

Author SHA1 Message Date
Morten Olsen
73df3ba0a6 ci: add NPM publish 2024-01-12 14:42:52 +01:00
67 changed files with 118 additions and 837 deletions

View File

@@ -1,5 +0,0 @@
{
"image": "mcr.microsoft.com/devcontainers/universal:2",
"features": {
}
}

View File

@@ -1,27 +0,0 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: bug
assignees: ''
---
**Describe the bug**
A clear and concise description of what the bug is.
**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Additional context**
Add any other context about the problem here.

View File

@@ -1,20 +0,0 @@
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: enhancement
assignees: ''
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.

View File

@@ -82,8 +82,6 @@ jobs:
uses: docker/metadata-action@9ec57ed1fcdbf14dcef7dfbe97b2010124a938b7 uses: docker/metadata-action@9ec57ed1fcdbf14dcef7dfbe97b2010124a938b7
with: with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
latest
- name: Build and push Docker image - name: Build and push Docker image
uses: docker/build-push-action@f2a1d5e99d037542a71f64918e516c093c6f3fc4 uses: docker/build-push-action@f2a1d5e99d037542a71f64918e516c093c6f3fc4
with: with:
@@ -119,7 +117,7 @@ jobs:
pnpm install pnpm install
git config user.name "Github Actions Bot" git config user.name "Github Actions Bot"
git config user.email "<>" git config user.email "<>"
node scripts/set-version.mjs $(git describe --tag --abbrev=0) node scripts/set-version.ts $(git describe --tag --abbrev=0)
pnpm publish -r --publish-branch main --access public --no-git-checks pnpm publish -r --publish-branch main --access public --no-git-checks
env: env:
NPM_TOKEN: ${{ secrets.NPM_TOKEN }} NPM_TOKEN: ${{ secrets.NPM_TOKEN }}

View File

@@ -1,43 +0,0 @@
# Code of Conduct
## Our Pledge
In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
## Our Standards
Examples of behavior that contributes to creating a positive environment include:
- Using welcoming and inclusive language
- Being respectful of differing viewpoints and experiences
- Gracefully accepting constructive criticism
- Focusing on what is best for the community
- Showing empathy towards other community members
Examples of unacceptable behavior by participants include:
- The use of sexualized language or imagery and unwelcome sexual attention or advances
- Trolling, insulting/derogatory comments, and personal or political attacks
- Public or private harassment
- Publishing others' private information, such as a physical or electronic address, without explicit permission
- Other conduct which could reasonably be considered inappropriate in a professional setting
## Our Responsibilities
Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned with this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
## Scope
This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at s56gkgkq@void.black. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org), version 1.4, available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html

View File

@@ -1,59 +0,0 @@
# Contributing to mini loader
First off, thank you for considering contributing to mini loader! It's people like you that make mini loader such a great tool.
## Code of Conduct
This project and everyone participating in it is governed by the [mini loader Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable behavior to s56gkgkq@void.black.
## How Can I Contribute?
### Reporting Bugs
This section guides you through submitting a bug report for mini loader. Following these guidelines helps maintainers and the community understand your report, reproduce the behavior, and find related reports.
**Before Submitting A Bug Report**
- Ensure the bug was not already reported by searching on GitHub under [Issues](https://github.com/morten-olsen/mini-loader/issues).
- If you're unable to find an open issue addressing the problem, open a new one. Be sure to include a title and clear description, as much relevant information as possible, and a code sample or an executable test case demonstrating the expected behavior that is not occurring.
### Suggesting Enhancements
This section guides you through submitting an enhancement suggestion for mini loader, including completely new features and minor improvements to existing functionality.
**Before Submitting An Enhancement Suggestion**
- Check if the enhancement has already been suggested under [Issues](https://github.com/morten-olsen/mini-loader/issues).
- If it hasn't, create a new issue and provide a concise description of the enhancement with as much detail as possible.
### Your First Code Contribution
Unsure where to begin contributing to mini loader? You can start by looking through `beginner` and `help-wanted` issues:
- Beginner issues - issues which should only require a few lines of code, and a test or two.
- Help wanted issues - issues which should be a bit more involved than `beginner` issues.
### Pull Requests
- Fill in the required template
- Do not include issue numbers in the PR title
- Follow the coding style used throughout the project
- Include appropriate test coverage. New features should include new tests.
- Document new code based on the [Documentation Styleguide](#documentation-styleguide)
## Documentation Styleguide
Use this style guide for documentation:
- Use Markdown
- Reference methods and classes in markdown backticks. For example, `ClassName.methodName`
- Document new code or add comments in code to explain parts that might be confusing.
## Use a Consistent Coding Style
* 2 spaces for indentation rather than tabs
* You can try running `pnpm run test:lint` for style unification
## License
By contributing to mini loader, you agree that your contributions will be licensed under its GPL-3 License.

View File

@@ -1,6 +1,10 @@
![banner](./assets/banner.png) <p>
<center>
<img src="./assets/logo.png" width="300" height="300" />
</center>
</p>
# Welcome to Mini Loader! 🌐 # Welcome to Mini Loader! 🌐
@@ -22,11 +26,9 @@ Also see [anti-features and limitations](./docs/anti-features.md)
Get up and running with mini loader in just a few steps: Get up and running with mini loader in just a few steps:
1. **Install the CLI**: `npm install -g @morten-olsen/mini-loader-cli` 1. **Install the CLI**: `npm install -g @morten-olsen/mini-loader-cli`
2. **Deploy the Server**: `docker run -p 4500:4500 -name mini-loader ghcr.io/morten-olsen/mini-loader`. 2. **Deploy the Server**: `docker run -p 4500:4500 -n mini-loader ghcr.io/morten-olsen/mini-loader:main`.
3. **Get your access token**: `docker exec mini-loader mini-loader-server create-token` 3. **Push Your First Load**: `mini-loader loads push script.mjs -r -i first`
4. **Login**: `mini-loader auth login http://localhost:4500` 3. **See the logs**: `mini-loader logs ls -l first`
5. **Push Your First Load**: `mini-loader loads push script.mjs -r -i first`
6. **See the logs**: `mini-loader logs ls -l first`
For a detailed guide on getting started, please refer to the [Getting Started Tutorial](./docs/getting-started.md). For a detailed guide on getting started, please refer to the [Getting Started Tutorial](./docs/getting-started.md).

Binary file not shown.

Before

Width:  |  Height:  |  Size: 418 KiB

View File

@@ -5,9 +5,5 @@ services:
build: build:
context: . context: .
dockerfile: ./docker/Dockerfile dockerfile: ./docker/Dockerfile
volumes:
- data:/app/data
ports: ports:
- 4500:4500 - 4500:4500
volumes:
data:

View File

@@ -27,7 +27,6 @@ COPY --from=builder /app/out/full/ .
RUN pnpm turbo run build --filter=@morten-olsen/mini-loader-server RUN pnpm turbo run build --filter=@morten-olsen/mini-loader-server
FROM base AS runner FROM base AS runner
RUN apk add --no-cache jq curl
WORKDIR /app WORKDIR /app
# Don't run production as root # Don't run production as root
@@ -39,12 +38,5 @@ RUN chmod +x /entrypoint.sh
COPY --from=installer /app . COPY --from=installer /app .
EXPOSE 4500 EXPOSE 4500
VOLUME /app/data
HEALTHCHECK \
--interval=10s \
--start-period=10s \
CMD curl -f http://localhost:4500/health || exit 1
ENTRYPOINT ["/entrypoint.sh"] ENTRYPOINT ["/entrypoint.sh"]
CMD ["mini-loader-server", "start"] CMD ["mini-loader-server", "start"]

View File

@@ -8,7 +8,7 @@ This guide will help you quickly set up and run a mini loader server using Docke
To begin, let's deploy the mini loader container. Run the following command in your terminal: To begin, let's deploy the mini loader container. Run the following command in your terminal:
```bash ```bash
docker run -p 4500:4500 -n mini-loader ghcr.io/morten-olsen/mini-loader:latest docker run -p 4500:4500 -n mini-loader ghcr.io/morten-olsen/mini-loader:main
``` ```
This command downloads the latest mini loader image and runs it, exposing port 4500. This command downloads the latest mini loader image and runs it, exposing port 4500.

View File

@@ -1,7 +1,7 @@
{ {
"name": "@morten-olsen/mini-loader-repo", "name": "@morten-olsen/mini-loader-repo",
"private": "true", "private": "true",
"license": "GPL-3.0", "license": "GPL-3.0-or-later",
"packageManager": "pnpm@8.10.4", "packageManager": "pnpm@8.10.4",
"version": "1.0.0", "version": "1.0.0",
"scripts": { "scripts": {
@@ -23,10 +23,5 @@
"@pnpm/find-workspace-packages": "^6.0.9", "@pnpm/find-workspace-packages": "^6.0.9",
"@types/node": "^20.10.8", "@types/node": "^20.10.8",
"ts-node": "^10.9.2" "ts-node": "^10.9.2"
},
"homepage": "https://github.com/morten-olsen/mini-loader",
"repository": {
"type": "git",
"url": "https://github.com/morten-olsen/mini-loader"
} }
} }

View File

@@ -1 +0,0 @@
[Go to documentation](https://github.com/morten-olsen/mini-loader)

View File

@@ -3,7 +3,7 @@
"version": "1.0.0", "version": "1.0.0",
"main": "./dist/esm/index.js", "main": "./dist/esm/index.js",
"types": "./dist/esm/index.d.ts", "types": "./dist/esm/index.d.ts",
"license": "GPL-3.0", "license": "GPL-3.0-or-later",
"bin": { "bin": {
"mini-loader": "./bin/index.mjs" "mini-loader": "./bin/index.mjs"
}, },
@@ -20,7 +20,6 @@
} }
}, },
"dependencies": { "dependencies": {
"@morten-olsen/mini-loader-runner": "workspace:^",
"@rollup/plugin-auto-install": "^3.0.5", "@rollup/plugin-auto-install": "^3.0.5",
"@rollup/plugin-commonjs": "^25.0.7", "@rollup/plugin-commonjs": "^25.0.7",
"@rollup/plugin-json": "^6.1.0", "@rollup/plugin-json": "^6.1.0",
@@ -29,7 +28,6 @@
"@rollup/plugin-sucrase": "^5.0.2", "@rollup/plugin-sucrase": "^5.0.2",
"@trpc/client": "^10.45.0", "@trpc/client": "^10.45.0",
"commander": "^11.1.0", "commander": "^11.1.0",
"dotenv": "^16.3.1",
"env-paths": "^3.0.0", "env-paths": "^3.0.0",
"inquirer": "^9.2.12", "inquirer": "^9.2.12",
"ora": "^8.0.1", "ora": "^8.0.1",
@@ -40,13 +38,9 @@
}, },
"devDependencies": { "devDependencies": {
"@morten-olsen/mini-loader-configs": "workspace:^", "@morten-olsen/mini-loader-configs": "workspace:^",
"@morten-olsen/mini-loader-runner": "workspace:^",
"@morten-olsen/mini-loader-server": "workspace:^", "@morten-olsen/mini-loader-server": "workspace:^",
"@types/inquirer": "^9.0.7", "@types/inquirer": "^9.0.7",
"typescript": "^5.3.3" "typescript": "^5.3.3"
},
"homepage": "https://github.com/morten-olsen/mini-loader",
"repository": {
"type": "git",
"url": "https://github.com/morten-olsen/mini-loader"
} }
} }

View File

@@ -17,12 +17,12 @@ const bundle = async ({ entry, autoInstall }: BundleOptions) => {
const entryFile = resolve(entry); const entryFile = resolve(entry);
const codeBundler = await rollup({ const codeBundler = await rollup({
plugins: [ plugins: [
fix(json)(),
fix(sucrase)({ fix(sucrase)({
transforms: ['typescript', 'jsx'], transforms: ['typescript', 'jsx'],
}), }),
...[autoInstall ? fix(auto) : []], ...[autoInstall ? fix(auto) : []],
nodeResolve({ preferBuiltins: true, extensions: ['.js', '.jsx', '.ts', '.tsx'] }), nodeResolve({ extensions: ['.js', '.jsx', '.ts', '.tsx'] }),
fix(json)(),
fix(commonjs)({ include: /node_modules/ }), fix(commonjs)({ include: /node_modules/ }),
], ],
input: entryFile, input: entryFile,

View File

@@ -2,7 +2,6 @@ import { createTRPCProxyClient, httpBatchLink } from '@trpc/client';
import superjson from 'superjson'; import superjson from 'superjson';
import type { Runtime } from '@morten-olsen/mini-loader-server'; import type { Runtime } from '@morten-olsen/mini-loader-server';
import type { RootRouter } from '@morten-olsen/mini-loader-server'; import type { RootRouter } from '@morten-olsen/mini-loader-server';
import pkg from '../../package.json';
import { Context } from '../context/context.js'; import { Context } from '../context/context.js';
const createClient = (context: Context) => { const createClient = (context: Context) => {
@@ -15,7 +14,6 @@ const createClient = (context: Context) => {
httpBatchLink({ httpBatchLink({
url: `${context.host}/trpc`, url: `${context.host}/trpc`,
headers: { headers: {
'x-version': pkg.version,
authorization: `Bearer ${context.token}`, authorization: `Bearer ${context.token}`,
}, },
}), }),

View File

@@ -2,7 +2,6 @@ import { Command } from 'commander';
import { createClient } from '../../client/client.js'; import { createClient } from '../../client/client.js';
import { step } from '../../utils/step.js'; import { step } from '../../utils/step.js';
import { Context } from '../../context/context.js'; import { Context } from '../../context/context.js';
import { Config } from '../../config/config.js';
const list = new Command('list'); const list = new Command('list');
@@ -22,8 +21,7 @@ list
.option('-a, --limit <limit>', 'Limit', '1000') .option('-a, --limit <limit>', 'Limit', '1000')
.action(async () => { .action(async () => {
const { runId, loadId, offset, limit } = list.opts(); const { runId, loadId, offset, limit } = list.opts();
const config = new Config(); const context = new Context();
const context = new Context(config.context);
const client = await step('Connecting to server', async () => { const client = await step('Connecting to server', async () => {
return createClient(context); return createClient(context);
}); });

View File

@@ -4,7 +4,6 @@ import { step } from '../../utils/step.js';
import { Context } from '../../context/context.js'; import { Context } from '../../context/context.js';
import { dirname, resolve } from 'path'; import { dirname, resolve } from 'path';
import { mkdir, writeFile } from 'fs/promises'; import { mkdir, writeFile } from 'fs/promises';
import { Config } from '../../config/config.js';
const pull = new Command('pull'); const pull = new Command('pull');
@@ -13,8 +12,7 @@ pull
.argument('<artifact-id>', 'Artifact ID') .argument('<artifact-id>', 'Artifact ID')
.argument('<file>', 'File to save') .argument('<file>', 'File to save')
.action(async (id, file) => { .action(async (id, file) => {
const config = new Config(); const context = new Context();
const context = new Context(config.context);
const target = resolve(file); const target = resolve(file);
const client = await step('Connecting to server', async () => { const client = await step('Connecting to server', async () => {
return createClient(context); return createClient(context);

View File

@@ -3,7 +3,6 @@ import { createClient } from '../../client/client.js';
import { step } from '../../utils/step.js'; import { step } from '../../utils/step.js';
import { Context } from '../../context/context.js'; import { Context } from '../../context/context.js';
import inquirer from 'inquirer'; import inquirer from 'inquirer';
import { Config } from '../../config/config.js';
const remove = new Command('remove'); const remove = new Command('remove');
@@ -23,8 +22,7 @@ remove
.option('-a, --limit <limit>', 'Limit', '1000') .option('-a, --limit <limit>', 'Limit', '1000')
.action(async () => { .action(async () => {
const { runId, loadId, offset, limit } = remove.opts(); const { runId, loadId, offset, limit } = remove.opts();
const config = new Config(); const context = new Context();
const context = new Context(config.context);
const client = await step('Connecting to server', async () => { const client = await step('Connecting to server', async () => {
return createClient(context); return createClient(context);
}); });

View File

@@ -2,14 +2,12 @@ import { Command } from 'commander';
import inquerer from 'inquirer'; import inquerer from 'inquirer';
import { Context } from '../../context/context.js'; import { Context } from '../../context/context.js';
import { step } from '../../utils/step.js'; import { step } from '../../utils/step.js';
import { Config } from '../../config/config.js';
const login = new Command('login'); const login = new Command('login');
login.description('Login to your account'); login.description('Login to your account');
login.action(async () => { login.action(async () => {
const config = new Config(); const context = new Context();
const context = new Context(config.context);
const { host, token } = await inquerer.prompt([ const { host, token } = await inquerer.prompt([
{ {
type: 'input', type: 'input',

View File

@@ -1,10 +0,0 @@
import { Command } from 'commander';
import { Config } from '../../config/config.js';
const current = new Command('current');
current.action(async () => {
const config = new Config();
console.log(config.context);
});
export { current };

View File

@@ -1,11 +0,0 @@
import { Command } from 'commander';
import { Context } from '../../context/context.js';
const list = new Command('list');
list.alias('ls').description('List contexts');
list.action(async () => {
const contexts = await Context.list();
console.table(contexts);
});
export { list };

View File

@@ -1,12 +0,0 @@
import { Command } from 'commander';
import { list } from './contexts.list.js';
import { use } from './contexts.use.js';
import { current } from './contexts.current.js';
const contexts = new Command('contexts');
contexts.description('Manage contexts');
contexts.addCommand(list);
contexts.addCommand(use);
contexts.addCommand(current);
export { contexts };

View File

@@ -1,11 +0,0 @@
import { Command } from 'commander';
import { Config } from '../../config/config.js';
const use = new Command('use');
use.argument('<name>').action(async (name) => {
const config = new Config();
await config.setContext(name);
});
export { use };

View File

@@ -2,7 +2,6 @@ import { Command } from 'commander';
import { createClient } from '../../client/client.js'; import { createClient } from '../../client/client.js';
import { step } from '../../utils/step.js'; import { step } from '../../utils/step.js';
import { Context } from '../../context/context.js'; import { Context } from '../../context/context.js';
import { Config } from '../../config/config.js';
const list = new Command('list'); const list = new Command('list');
@@ -10,8 +9,7 @@ list
.alias('ls') .alias('ls')
.description('List loads') .description('List loads')
.action(async () => { .action(async () => {
const config = new Config(); const context = new Context();
const context = new Context(config.context);
const client = await step('Connecting to server', async () => { const client = await step('Connecting to server', async () => {
return createClient(context); return createClient(context);
}); });

View File

@@ -4,7 +4,6 @@ import { createClient } from '../../client/client.js';
import { bundle } from '../../bundler/bundler.js'; import { bundle } from '../../bundler/bundler.js';
import { step } from '../../utils/step.js'; import { step } from '../../utils/step.js';
import { Context } from '../../context/context.js'; import { Context } from '../../context/context.js';
import { Config } from '../../config/config.js';
const push = new Command('push'); const push = new Command('push');
@@ -16,8 +15,7 @@ push
.option('-ai, --auto-install', 'Auto install dependencies', false) .option('-ai, --auto-install', 'Auto install dependencies', false)
.action(async (script) => { .action(async (script) => {
const opts = push.opts(); const opts = push.opts();
const config = new Config(); const context = new Context();
const context = new Context(config.context);
const location = resolve(script); const location = resolve(script);
const client = await step('Connecting to server', async () => { const client = await step('Connecting to server', async () => {
return createClient(context); return createClient(context);
@@ -25,7 +23,7 @@ push
const code = await step('Bundling', async () => { const code = await step('Bundling', async () => {
return await bundle({ entry: location, autoInstall: opts.autoInstall }); return await bundle({ entry: location, autoInstall: opts.autoInstall });
}); });
const id = await step(`Creating load ${(code.length / 1024).toFixed(0)}`, async () => { const id = await step('Creating load', async () => {
return await client.loads.set.mutate({ return await client.loads.set.mutate({
id: opts.id, id: opts.id,
name: opts.name, name: opts.name,
@@ -34,10 +32,9 @@ push
}); });
console.log('created load with id', id); console.log('created load with id', id);
if (opts.run) { if (opts.run) {
const runId = await step('Creating run', async () => { await step('Creating run', async () => {
return await client.runs.create.mutate({ loadId: id }); await client.runs.create.mutate({ loadId: id });
}); });
console.log('created run with id', runId);
} }
}); });

View File

@@ -3,7 +3,6 @@ import { resolve } from 'path';
import { run as runLoad } from '@morten-olsen/mini-loader-runner'; import { run as runLoad } from '@morten-olsen/mini-loader-runner';
import { bundle } from '../../bundler/bundler.js'; import { bundle } from '../../bundler/bundler.js';
import { step } from '../../utils/step.js'; import { step } from '../../utils/step.js';
import { readSecrets } from './local.utils.js';
const run = new Command('run'); const run = new Command('run');
@@ -13,14 +12,12 @@ run
.action(async (script) => { .action(async (script) => {
const location = resolve(script); const location = resolve(script);
const { autoInstall } = run.opts(); const { autoInstall } = run.opts();
const secrets = await readSecrets();
const code = await step('Bundling', async () => { const code = await step('Bundling', async () => {
return await bundle({ entry: location, autoInstall }); return await bundle({ entry: location, autoInstall });
}); });
const { promise, emitter } = await runLoad({ const { promise, emitter } = await runLoad({
script: code, script: code,
secrets,
}); });
emitter.addListener('message', (message) => { emitter.addListener('message', (message) => {
switch (message.type) { switch (message.type) {

View File

@@ -1,25 +0,0 @@
import dotenv from 'dotenv';
import { existsSync } from 'fs';
import { readFile } from 'fs/promises';
import { join } from 'path';
const ENV_PREFIX = 'ML_S_';
const readSecrets = async () => {
let secretLocation = join(process.cwd(), '.secret');
let secrets: Record<string, string> = {};
if (existsSync(secretLocation)) {
const content = await readFile(secretLocation, 'utf-8');
secrets = dotenv.parse(content);
}
for (const key in process.env) {
if (key.startsWith(ENV_PREFIX)) {
secrets[key.replace(ENV_PREFIX, '')] = process.env[key]!;
}
}
return secrets;
};
export { readSecrets };

View File

@@ -2,7 +2,6 @@ import { Command } from 'commander';
import { createClient } from '../../client/client.js'; import { createClient } from '../../client/client.js';
import { step } from '../../utils/step.js'; import { step } from '../../utils/step.js';
import { Context } from '../../context/context.js'; import { Context } from '../../context/context.js';
import { Config } from '../../config/config.js';
const list = new Command('list'); const list = new Command('list');
@@ -24,8 +23,7 @@ list
.option('-s, --sort <order>', 'Sort', 'desc') .option('-s, --sort <order>', 'Sort', 'desc')
.action(async () => { .action(async () => {
const { runId, loadId, severities, offset, limit, order } = list.opts(); const { runId, loadId, severities, offset, limit, order } = list.opts();
const config = new Config(); const context = new Context();
const context = new Context(config.context);
const client = await step('Connecting to server', async () => { const client = await step('Connecting to server', async () => {
return createClient(context); return createClient(context);
}); });

View File

@@ -3,7 +3,6 @@ import { createClient } from '../../client/client.js';
import { step } from '../../utils/step.js'; import { step } from '../../utils/step.js';
import { Context } from '../../context/context.js'; import { Context } from '../../context/context.js';
import inquirer from 'inquirer'; import inquirer from 'inquirer';
import { Config } from '../../config/config.js';
const remove = new Command('remove'); const remove = new Command('remove');
@@ -25,8 +24,7 @@ remove
.option('-s, --sort <order>', 'Sort', 'desc') .option('-s, --sort <order>', 'Sort', 'desc')
.action(async () => { .action(async () => {
const { runId, loadId, severities, offset, limit, order } = remove.opts(); const { runId, loadId, severities, offset, limit, order } = remove.opts();
const config = new Config(); const context = new Context();
const context = new Context(config.context);
const client = await step('Connecting to server', async () => { const client = await step('Connecting to server', async () => {
return createClient(context); return createClient(context);
}); });

View File

@@ -2,7 +2,6 @@ import { Command } from 'commander';
import { createClient } from '../../client/client.js'; import { createClient } from '../../client/client.js';
import { step } from '../../utils/step.js'; import { step } from '../../utils/step.js';
import { Context } from '../../context/context.js'; import { Context } from '../../context/context.js';
import { Config } from '../../config/config.js';
const create = new Command('create'); const create = new Command('create');
@@ -10,8 +9,7 @@ create
.description('Create a new run') .description('Create a new run')
.argument('load-id', 'Load ID') .argument('load-id', 'Load ID')
.action(async (loadId) => { .action(async (loadId) => {
const config = new Config(); const context = new Context();
const context = new Context(config.context);
const client = await step('Connecting to server', async () => { const client = await step('Connecting to server', async () => {
return createClient(context); return createClient(context);
}); });

View File

@@ -2,7 +2,6 @@ import { Command } from 'commander';
import { createClient } from '../../client/client.js'; import { createClient } from '../../client/client.js';
import { step } from '../../utils/step.js'; import { step } from '../../utils/step.js';
import { Context } from '../../context/context.js'; import { Context } from '../../context/context.js';
import { Config } from '../../config/config.js';
const list = new Command('list'); const list = new Command('list');
@@ -11,8 +10,7 @@ list
.description('Find a run') .description('Find a run')
.argument('[load-id]', 'Load ID') .argument('[load-id]', 'Load ID')
.action(async (loadId) => { .action(async (loadId) => {
const config = new Config(); const context = new Context();
const context = new Context(config.context);
const client = await step('Connecting to server', async () => { const client = await step('Connecting to server', async () => {
return createClient(context); return createClient(context);
}); });

View File

@@ -1,59 +0,0 @@
import { Command } from 'commander';
import { createClient } from '../../client/client.js';
import { step } from '../../utils/step.js';
import { Context } from '../../context/context.js';
import inquirer from 'inquirer';
import { Config } from '../../config/config.js';
const remove = new Command('remove');
const toInt = (value?: string) => {
if (!value) {
return undefined;
}
return parseInt(value, 10);
};
remove
.alias('ls')
.description('List logs')
.option('-l, --load-id <loadId>', 'Load ID')
.option('-o, --offset <offset>', 'Offset')
.option('-a, --limit <limit>', 'Limit', '1000')
.action(async () => {
const { loadId, offset, limit } = remove.opts();
const config = new Config();
const context = new Context(config.context);
const client = await step('Connecting to server', async () => {
return createClient(context);
});
const response = await step('Preparing to delete', async () => {
return await client.runs.prepareRemove.query({
loadId,
offset: toInt(offset),
limit: toInt(limit),
});
});
if (!response.ids.length) {
console.log('No logs to delete');
return;
}
const { confirm } = await inquirer.prompt([
{
type: 'confirm',
name: 'confirm',
message: `Are you sure you want to delete ${response.ids.length} logs?`,
},
]);
if (!confirm) {
return;
}
await step('Deleting artifacts', async () => {
await client.runs.remove.mutate(response);
});
});
export { remove };

View File

@@ -1,23 +0,0 @@
import { Command } from 'commander';
import { createClient } from '../../client/client.js';
import { step } from '../../utils/step.js';
import { Context } from '../../context/context.js';
import { Config } from '../../config/config.js';
const terminate = new Command('terminate');
terminate
.description('Terminate an in progress run')
.argument('run-id', 'Run ID')
.action(async (runId) => {
const config = new Config();
const context = new Context(config.context);
const client = await step('Connecting to server', async () => {
return createClient(context);
});
await step('Terminating run', async () => {
await client.runs.terminate.mutate(runId);
});
});
export { terminate };

View File

@@ -1,14 +1,8 @@
import { Command } from 'commander'; import { Command } from 'commander';
import { create } from './runs.create.js'; import { create } from './runs.create.js';
import { list } from './runs.list.js'; import { list } from './runs.list.js';
import { remove } from './runs.remove.js';
import { terminate } from './runs.terminate.js';
const runs = new Command('runs'); const runs = new Command('runs');
runs.description('Manage runs'); runs.description('Manage runs').addCommand(create).addCommand(list);
runs.addCommand(create);
runs.addCommand(list);
runs.addCommand(remove);
runs.addCommand(terminate);
export { runs }; export { runs };

View File

@@ -2,7 +2,6 @@ import { Command } from 'commander';
import { createClient } from '../../client/client.js'; import { createClient } from '../../client/client.js';
import { step } from '../../utils/step.js'; import { step } from '../../utils/step.js';
import { Context } from '../../context/context.js'; import { Context } from '../../context/context.js';
import { Config } from '../../config/config.js';
const list = new Command('list'); const list = new Command('list');
@@ -20,8 +19,7 @@ list
.option('-a, --limit <limit>', 'Limit', '1000') .option('-a, --limit <limit>', 'Limit', '1000')
.action(async () => { .action(async () => {
const { offset, limit } = list.opts(); const { offset, limit } = list.opts();
const config = new Config(); const context = new Context();
const context = new Context(config.context);
const client = await step('Connecting to server', async () => { const client = await step('Connecting to server', async () => {
return createClient(context); return createClient(context);
}); });

View File

@@ -2,7 +2,6 @@ import { Command } from 'commander';
import { createClient } from '../../client/client.js'; import { createClient } from '../../client/client.js';
import { step } from '../../utils/step.js'; import { step } from '../../utils/step.js';
import { Context } from '../../context/context.js'; import { Context } from '../../context/context.js';
import { Config } from '../../config/config.js';
const remove = new Command('remove'); const remove = new Command('remove');
@@ -10,8 +9,7 @@ remove
.alias('rm') .alias('rm')
.argument('<id>') .argument('<id>')
.action(async (id) => { .action(async (id) => {
const config = new Config(); const context = new Context();
const context = new Context(config.context);
const client = await step('Connecting to server', async () => { const client = await step('Connecting to server', async () => {
return createClient(context); return createClient(context);
}); });

View File

@@ -2,7 +2,6 @@ import { Command } from 'commander';
import { createClient } from '../../client/client.js'; import { createClient } from '../../client/client.js';
import { step } from '../../utils/step.js'; import { step } from '../../utils/step.js';
import { Context } from '../../context/context.js'; import { Context } from '../../context/context.js';
import { Config } from '../../config/config.js';
const set = new Command('set'); const set = new Command('set');
@@ -10,8 +9,7 @@ set
.argument('<id>') .argument('<id>')
.argument('[value]') .argument('[value]')
.action(async (id, value) => { .action(async (id, value) => {
const config = new Config(); const context = new Context();
const context = new Context(config.context);
const client = await step('Connecting to server', async () => { const client = await step('Connecting to server', async () => {
return createClient(context); return createClient(context);
}); });

View File

@@ -1,44 +0,0 @@
import envPaths from 'env-paths';
import { existsSync, readFileSync, writeFileSync } from 'fs';
import { mkdir } from 'fs/promises';
import { join, dirname } from 'path';
type ConfigValues = {
context?: string;
};
class Config {
#location: string;
#config?: ConfigValues;
constructor() {
const paths = envPaths('mini-loader');
this.#location = join(paths.config, 'config.json');
if (existsSync(this.#location)) {
this.#config = JSON.parse(readFileSync(this.#location, 'utf-8'));
}
}
public get context() {
return this.#config?.context || 'default';
}
public setContext = (context: string) => {
this.#config = {
...(this.#config || {}),
context,
};
this.save();
};
public save = async () => {
if (!this.#config) {
return;
}
const json = JSON.stringify(this.#config);
mkdir(dirname(this.#location), { recursive: true });
writeFileSync(this.#location, json);
};
}
export { Config };

View File

@@ -1,7 +1,7 @@
import envPaths from 'env-paths'; import envPaths from 'env-paths';
import { existsSync, readFileSync, writeFileSync } from 'fs'; import { existsSync, readFileSync, writeFileSync } from 'fs';
import { mkdir, readdir } from 'fs/promises'; import { mkdir } from 'fs/promises';
import { dirname, join } from 'path'; import { dirname } from 'path';
type ContextValues = { type ContextValues = {
host: string; host: string;
@@ -12,9 +12,9 @@ class Context {
#location: string; #location: string;
#config?: ContextValues; #config?: ContextValues;
constructor(name: string) { constructor() {
const paths = envPaths('mini-loader'); const paths = envPaths('dws');
this.#location = join(paths.config, 'contexts', name); this.#location = paths.config;
if (existsSync(this.#location)) { if (existsSync(this.#location)) {
this.#config = JSON.parse(readFileSync(this.#location, 'utf-8')); this.#config = JSON.parse(readFileSync(this.#location, 'utf-8'));
} }
@@ -45,15 +45,6 @@ class Context {
mkdir(dirname(this.#location), { recursive: true }); mkdir(dirname(this.#location), { recursive: true });
writeFileSync(this.#location, json); writeFileSync(this.#location, json);
}; };
public static list = async () => {
const paths = envPaths('mini-loader');
const location = join(paths.config, 'contexts');
if (!existsSync(location)) {
return [];
}
return await readdir(location);
};
} }
export { Context }; export { Context };

View File

@@ -1,5 +1,4 @@
import { Command, program } from 'commander'; import { program } from 'commander';
import pkg from '../package.json';
import { loads } from './commands/loads/loads.js'; import { loads } from './commands/loads/loads.js';
import { runs } from './commands/runs/runs.js'; import { runs } from './commands/runs/runs.js';
import { logs } from './commands/logs/logs.js'; import { logs } from './commands/logs/logs.js';
@@ -7,7 +6,6 @@ import { artifacts } from './commands/artifacts/artifacts.js';
import { secrets } from './commands/secrets/secrets.js'; import { secrets } from './commands/secrets/secrets.js';
import { local } from './commands/local/local.js'; import { local } from './commands/local/local.js';
import { auth } from './commands/auth/auth.js'; import { auth } from './commands/auth/auth.js';
import { contexts } from './commands/contexts/contexts.js';
program.addCommand(loads); program.addCommand(loads);
program.addCommand(runs); program.addCommand(runs);
@@ -16,14 +14,5 @@ program.addCommand(artifacts);
program.addCommand(secrets); program.addCommand(secrets);
program.addCommand(local); program.addCommand(local);
program.addCommand(auth); program.addCommand(auth);
program.addCommand(contexts);
program.version(pkg.version);
const version = new Command('version');
version.action(() => {
console.log(pkg.version);
});
program.addCommand(version);
await program.parseAsync(); await program.parseAsync();

View File

@@ -9,10 +9,5 @@
}, },
"keywords": [], "keywords": [],
"author": "", "author": "",
"license": "GPL-3.0", "license": "GPL-3.0-or-later"
"homepage": "https://github.com/morten-olsen/mini-loader", }
"repository": {
"type": "git",
"url": "https://github.com/morten-olsen/mini-loader"
}
}

View File

@@ -8,7 +8,6 @@
"sourceMap": true, "sourceMap": true,
"esModuleInterop": true, "esModuleInterop": true,
"strict": true, "strict": true,
"resolveJsonModule": true,
"allowSyntheticDefaultImports": true, "allowSyntheticDefaultImports": true,
"jsx": "react" "jsx": "react"
}, },

View File

@@ -1,7 +1,7 @@
{ {
"name": "@morten-olsen/mini-loader-examples", "name": "@morten-olsen/mini-loader-examples",
"version": "1.0.0", "version": "1.0.0",
"license": "GPL-3.0", "license": "GPL-3.0-or-later",
"private": true, "private": true,
"main": "./dist/esm/index.js", "main": "./dist/esm/index.js",
"types": "./dist/esm/index.d.ts", "types": "./dist/esm/index.d.ts",
@@ -18,18 +18,10 @@
} }
}, },
"devDependencies": { "devDependencies": {
"@morten-olsen/mini-loader": "workspace:^",
"@morten-olsen/mini-loader-cli": "workspace:^",
"@morten-olsen/mini-loader-configs": "workspace:^", "@morten-olsen/mini-loader-configs": "workspace:^",
"@morten-olsen/mini-loader-cli": "workspace:^",
"@morten-olsen/mini-loader": "workspace:^",
"@types/node": "^20.10.8", "@types/node": "^20.10.8",
"typescript": "^5.3.3" "typescript": "^5.3.3"
},
"homepage": "https://github.com/morten-olsen/mini-loader",
"repository": {
"type": "git",
"url": "https://github.com/morten-olsen/mini-loader"
},
"dependencies": {
"fastify": "^4.25.2"
} }
} }

View File

@@ -1,12 +0,0 @@
import { http } from '@morten-olsen/mini-loader';
import fastify from 'fastify';
const server = fastify();
server.all('*', async (req) => {
return req.url;
});
server.listen({
path: http.getPath(),
});

View File

@@ -3,7 +3,6 @@ import { artifacts, logger } from '@morten-olsen/mini-loader';
const run = async () => { const run = async () => {
await logger.info('Hello world'); await logger.info('Hello world');
await artifacts.create('foo', 'bar'); await artifacts.create('foo', 'bar');
process.exit(0);
}; };
run(); run();

View File

@@ -1 +0,0 @@
[Go to documentation](https://github.com/morten-olsen/mini-loader)

View File

@@ -1,7 +1,7 @@
{ {
"name": "@morten-olsen/mini-loader", "name": "@morten-olsen/mini-loader",
"version": "1.0.0", "version": "1.0.0",
"license": "GPL-3.0", "license": "GPL-3.0-or-later",
"main": "./dist/esm/index.js", "main": "./dist/esm/index.js",
"types": "./dist/esm/index.d.ts", "types": "./dist/esm/index.d.ts",
"scripts": { "scripts": {
@@ -20,10 +20,5 @@
"@morten-olsen/mini-loader-configs": "workspace:^", "@morten-olsen/mini-loader-configs": "workspace:^",
"@types/node": "^20.10.8", "@types/node": "^20.10.8",
"typescript": "^5.3.3" "typescript": "^5.3.3"
},
"homepage": "https://github.com/morten-olsen/mini-loader",
"repository": {
"type": "git",
"url": "https://github.com/morten-olsen/mini-loader"
} }
} }

View File

@@ -1,7 +0,0 @@
const getPath = () => process.env.HTTP_GATEWAY_PATH!;
const http = {
getPath,
};
export { http };

View File

@@ -8,4 +8,3 @@ export { logger } from './logger/logger.js';
export { artifacts } from './artifacts/artifacts.js'; export { artifacts } from './artifacts/artifacts.js';
export { input } from './input/input.js'; export { input } from './input/input.js';
export { secrets } from './secrets/secrets.js'; export { secrets } from './secrets/secrets.js';
export { http } from './http/http.js';

View File

@@ -1 +0,0 @@
[Go to documentation](https://github.com/morten-olsen/mini-loader)

View File

@@ -1,7 +1,7 @@
{ {
"name": "@morten-olsen/mini-loader-runner", "name": "@morten-olsen/mini-loader-runner",
"version": "1.0.0", "version": "1.0.0",
"license": "GPL-3.0", "license": "GPL-3.0-or-later",
"main": "./dist/esm/index.js", "main": "./dist/esm/index.js",
"types": "./dist/esm/index.d.ts", "types": "./dist/esm/index.d.ts",
"scripts": { "scripts": {
@@ -17,18 +17,13 @@
} }
}, },
"devDependencies": { "devDependencies": {
"@morten-olsen/mini-loader": "workspace:^",
"@morten-olsen/mini-loader-configs": "workspace:^", "@morten-olsen/mini-loader-configs": "workspace:^",
"@types/node": "^20.10.8", "@types/node": "^20.10.8",
"typescript": "^5.3.3" "typescript": "^5.3.3"
}, },
"dependencies": { "dependencies": {
"@morten-olsen/mini-loader": "workspace:^",
"eventemitter3": "^5.0.1", "eventemitter3": "^5.0.1",
"nanoid": "^5.0.4" "nanoid": "^5.0.4"
},
"homepage": "https://github.com/morten-olsen/mini-loader",
"repository": {
"type": "git",
"url": "https://github.com/morten-olsen/mini-loader"
} }
} }

View File

@@ -1,5 +1,17 @@
import { Worker } from 'worker_threads'; import { Worker } from 'worker_threads';
import { setup } from './setup/setup.js'; import os from 'os';
import { EventEmitter } from 'eventemitter3';
import { Event } from '@morten-olsen/mini-loader';
import { join } from 'path';
import { createServer } from 'http';
import { nanoid } from 'nanoid';
import { chmod, mkdir, rm, writeFile } from 'fs/promises';
type RunEvents = {
message: (event: Event) => void;
error: (error: Error) => void;
exit: () => void;
};
type RunOptions = { type RunOptions = {
script: string; script: string;
@@ -8,17 +20,44 @@ type RunOptions = {
}; };
const run = async ({ script, input, secrets }: RunOptions) => { const run = async ({ script, input, secrets }: RunOptions) => {
const info = await setup({ script, input, secrets }); const dataDir = join(os.tmpdir(), 'mini-loader', nanoid());
await mkdir(dataDir, { recursive: true });
await chmod(dataDir, 0o700);
const hostSocket = join(dataDir, 'host');
const server = createServer();
const inputLocation = join(dataDir, 'input');
const worker = new Worker(info.scriptLocation, { if (input) {
await writeFile(inputLocation, input);
}
const emitter = new EventEmitter<RunEvents>();
server.on('connection', (socket) => {
socket.on('data', (data) => {
const message = JSON.parse(data.toString());
emitter.emit('message', message);
});
});
server.listen(hostSocket);
const worker = new Worker(script, {
eval: true,
stdin: false, stdin: false,
stdout: false, stdout: false,
stderr: false, stderr: false,
env: info.env, env: {
HOST_SOCKET: hostSocket,
SECRETS: JSON.stringify(secrets),
INPUT_PATH: inputLocation,
},
workerData: {
input,
},
}); });
worker.stdout?.on('data', (data) => { worker.stdout?.on('data', (data) => {
info.emitter.emit('message', { emitter.emit('message', {
type: 'log', type: 'log',
payload: { payload: {
severity: 'info', severity: 'info',
@@ -28,7 +67,7 @@ const run = async ({ script, input, secrets }: RunOptions) => {
}); });
worker.stderr?.on('data', (data) => { worker.stderr?.on('data', (data) => {
info.emitter.emit('message', { emitter.emit('message', {
type: 'log', type: 'log',
payload: { payload: {
severity: 'error', severity: 'error',
@@ -39,24 +78,20 @@ const run = async ({ script, input, secrets }: RunOptions) => {
const promise = new Promise<void>((resolve, reject) => { const promise = new Promise<void>((resolve, reject) => {
worker.on('exit', async () => { worker.on('exit', async () => {
await info.teardown(); server.close();
await rm(dataDir, { recursive: true, force: true });
resolve(); resolve();
}); });
worker.on('error', async (error) => { worker.on('error', async (error) => {
server.close();
reject(error); reject(error);
}); });
}); });
return { return {
...info, emitter,
teardown: async () => {
worker.terminate();
},
promise, promise,
}; };
}; };
type RunInfo = Awaited<ReturnType<typeof run>>;
export type { RunInfo };
export { run }; export { run };

View File

@@ -1,71 +0,0 @@
import { join } from 'path';
import os from 'os';
import { nanoid } from 'nanoid';
import { chmod, mkdir, rm, writeFile } from 'fs/promises';
import { createServer } from 'net';
import { EventEmitter } from 'eventemitter3';
type SetupOptions = {
input?: Buffer | string;
script: string;
secrets?: Record<string, string>;
};
type RunEvents = {
message: (event: any) => void;
error: (error: Error) => void;
exit: () => void;
};
const setup = async (options: SetupOptions) => {
const { input, script, secrets } = options;
const emitter = new EventEmitter<RunEvents>();
const dataDir = join(os.tmpdir(), 'mini-loader', nanoid());
await mkdir(dataDir, { recursive: true });
await chmod(dataDir, 0o700);
const hostSocket = join(dataDir, 'host');
const httpGatewaySocket = join(dataDir, 'socket');
const server = createServer();
const inputLocation = join(dataDir, 'input');
const scriptLocation = join(dataDir, 'script.js');
if (input) {
await writeFile(inputLocation, input);
}
await writeFile(scriptLocation, script);
const env = {
HOST_SOCKET: hostSocket,
SECRETS: JSON.stringify(secrets || {}),
INPUT_PATH: inputLocation,
HTTP_GATEWAY_PATH: httpGatewaySocket,
};
const teardown = async () => {
server.close();
await rm(dataDir, { recursive: true, force: true });
};
server.on('connection', (socket) => {
socket.on('data', (data) => {
const message = JSON.parse(data.toString());
emitter.emit('message', message);
});
});
server.listen(hostSocket);
return {
env,
emitter,
teardown,
httpGatewaySocket,
scriptLocation,
hostSocket,
};
};
type Setup = Awaited<ReturnType<typeof setup>>;
export type { Setup };
export { setup };

View File

@@ -1 +0,0 @@
[Go to documentation](https://github.com/morten-olsen/mini-loader)

View File

@@ -1,7 +1,7 @@
{ {
"name": "@morten-olsen/mini-loader-server", "name": "@morten-olsen/mini-loader-server",
"version": "1.0.0", "version": "1.0.0",
"license": "GPL-3.0", "license": "GPL-3.0-or-later",
"main": "./dist/esm/index.js", "main": "./dist/esm/index.js",
"types": "./dist/esm/index.d.ts", "types": "./dist/esm/index.d.ts",
"bin": { "bin": {
@@ -27,7 +27,6 @@
"typescript": "^5.3.3" "typescript": "^5.3.3"
}, },
"dependencies": { "dependencies": {
"@fastify/reply-from": "^9.7.0",
"@trpc/client": "^10.45.0", "@trpc/client": "^10.45.0",
"@trpc/server": "^10.45.0", "@trpc/server": "^10.45.0",
"commander": "^11.1.0", "commander": "^11.1.0",
@@ -40,10 +39,5 @@
"sqlite3": "^5.1.7", "sqlite3": "^5.1.7",
"superjson": "^2.2.1", "superjson": "^2.2.1",
"zod": "^3.22.4" "zod": "^3.22.4"
},
"homepage": "https://github.com/morten-olsen/mini-loader",
"repository": {
"type": "git",
"url": "https://github.com/morten-olsen/mini-loader"
} }
} }

View File

@@ -1,34 +0,0 @@
import { FastifyPluginAsync } from 'fastify';
import FastifyReplyFrom from '@fastify/reply-from';
import { escape } from 'querystring';
import { Runtime } from '../runtime/runtime.js';
type Options = {
runtime: Runtime;
};
const gateway: FastifyPluginAsync<Options> = async (fastify, { runtime }) => {
await fastify.register(FastifyReplyFrom, {
http: {},
});
fastify.all('/gateway/*', (req, res) => {
const [runId, ...pathSegments] = (req.params as any)['*'].split('/').filter(Boolean);
const run = runtime.runner.getInstance(runId);
if (!run) {
res.statusCode = 404;
res.send({ error: 'Run not found' });
return;
}
const socketPath = run.run?.httpGatewaySocket;
if (!socketPath) {
res.statusCode = 404;
res.send({ error: 'No socket path to run' });
return;
}
const path = pathSegments.join('/');
res.from(`unix+http://${escape(socketPath)}/${path}`);
});
};
export { gateway };

View File

@@ -18,11 +18,7 @@ start.action(async () => {
const createToken = new Command('create-token'); const createToken = new Command('create-token');
createToken.action(async () => { createToken.action(async () => {
const runtime = await Runtime.create(); const runtime = await Runtime.create();
const token = await runtime.auth.createToken({ const token = await runtime.auth.createToken({});
policy: {
'*:*': ['*'],
},
});
console.log(token); console.log(token);
}); });

View File

@@ -3,7 +3,6 @@ import { EventEmitter } from 'eventemitter3';
import { Database } from '../../database/database.js'; import { Database } from '../../database/database.js';
import { CreateRunOptions, FindRunsOptions, UpdateRunOptions } from './runs.schemas.js'; import { CreateRunOptions, FindRunsOptions, UpdateRunOptions } from './runs.schemas.js';
import { LoadRepo } from '../loads/loads.js'; import { LoadRepo } from '../loads/loads.js';
import { createHash } from 'crypto';
type RunRepoEvents = { type RunRepoEvents = {
created: (args: { id: string; loadId: string }) => void; created: (args: { id: string; loadId: string }) => void;
@@ -19,22 +18,13 @@ type RunRepoOptions = {
class RunRepo extends EventEmitter<RunRepoEvents> { class RunRepo extends EventEmitter<RunRepoEvents> {
#options: RunRepoOptions; #options: RunRepoOptions;
#isReady: Promise<void>;
constructor(options: RunRepoOptions) { constructor(options: RunRepoOptions) {
super(); super();
this.#options = options; this.#options = options;
this.#isReady = this.#setup();
} }
#setup = async () => {
const { database } = this.#options;
const db = await database.instance;
await db('runs').update({ status: 'failed', error: 'server was shut down' }).where({ status: 'running' });
};
public getById = async (id: string) => { public getById = async (id: string) => {
await this.#isReady;
const { database } = this.#options; const { database } = this.#options;
const db = await database.instance; const db = await database.instance;
@@ -46,7 +36,6 @@ class RunRepo extends EventEmitter<RunRepoEvents> {
}; };
public getByLoadId = async (loadId: string) => { public getByLoadId = async (loadId: string) => {
await this.#isReady;
const { database } = this.#options; const { database } = this.#options;
const db = await database.instance; const db = await database.instance;
@@ -55,7 +44,6 @@ class RunRepo extends EventEmitter<RunRepoEvents> {
}; };
public find = async (options: FindRunsOptions) => { public find = async (options: FindRunsOptions) => {
await this.#isReady;
const { database } = this.#options; const { database } = this.#options;
const db = await database.instance; const db = await database.instance;
const query = db('runs').select(['id', 'status', 'startedAt', 'status', 'error', 'endedAt']); const query = db('runs').select(['id', 'status', 'startedAt', 'status', 'error', 'endedAt']);
@@ -74,41 +62,19 @@ class RunRepo extends EventEmitter<RunRepoEvents> {
return runs; return runs;
}; };
public prepareRemove = async (options: FindRunsOptions) => { public remove = async (options: FindRunsOptions) => {
await this.#isReady;
const { database } = this.#options; const { database } = this.#options;
const db = await database.instance; const db = await database.instance;
const query = db('runs').select('id'); const query = db('runs');
if (options.loadId) { if (options.loadId) {
query.where({ loadId: options.loadId }); query.where({ loadId: options.loadId });
} }
const result = await query; await query.del();
const ids = result.map((row) => row.id);
const token = ids.map((id) => Buffer.from(id).toString('base64')).join('|');
const hash = createHash('sha256').update(token).digest('hex');
return {
ids,
hash,
};
};
public remove = async (hash: string, ids: string[]) => {
const { database } = this.#options;
const db = await database.instance;
const token = ids.map((id) => Buffer.from(id).toString('base64')).join('|');
const actualHash = createHash('sha256').update(token).digest('hex');
if (hash !== actualHash) {
throw new Error('Invalid hash');
}
await db('runs').whereIn('id', ids).delete();
}; };
public started = async (id: string) => { public started = async (id: string) => {
await this.#isReady;
const { database } = this.#options; const { database } = this.#options;
const db = await database.instance; const db = await database.instance;
const current = await this.getById(id); const current = await this.getById(id);
@@ -126,7 +92,6 @@ class RunRepo extends EventEmitter<RunRepoEvents> {
}; };
public finished = async (id: string, options: UpdateRunOptions) => { public finished = async (id: string, options: UpdateRunOptions) => {
await this.#isReady;
const { database } = this.#options; const { database } = this.#options;
const db = await database.instance; const db = await database.instance;
const { loadId } = await this.getById(id); const { loadId } = await this.getById(id);
@@ -149,7 +114,6 @@ class RunRepo extends EventEmitter<RunRepoEvents> {
}; };
public create = async (options: CreateRunOptions) => { public create = async (options: CreateRunOptions) => {
await this.#isReady;
const { database, loads } = this.#options; const { database, loads } = this.#options;
const id = nanoid(); const id = nanoid();
const db = await database.instance; const db = await database.instance;

View File

@@ -1,4 +1,3 @@
import { z } from 'zod';
import { createRunSchema, findRunsSchema } from '../repos/repos.js'; import { createRunSchema, findRunsSchema } from '../repos/repos.js';
import { publicProcedure, router } from './router.utils.js'; import { publicProcedure, router } from './router.utils.js';
@@ -18,50 +17,17 @@ const find = publicProcedure.input(findRunsSchema).query(async ({ input, ctx })
return results; return results;
}); });
const prepareRemove = publicProcedure.input(findRunsSchema).query(async ({ input, ctx }) => { const remove = publicProcedure.input(findRunsSchema).mutation(async ({ input, ctx }) => {
const { runtime } = ctx; const { runtime } = ctx;
const { repos } = runtime; const { repos } = runtime;
const { runs } = repos; const { runs } = repos;
return await runs.prepareRemove(input); await runs.remove(input);
});
const remove = publicProcedure
.input(
z.object({
hash: z.string(),
ids: z.array(z.string()),
}),
)
.mutation(async ({ input, ctx }) => {
const { runtime } = ctx;
const { repos } = runtime;
const { runs } = repos;
for (const id of input.ids) {
const instance = runtime.runner.getInstance(id);
if (instance) {
await instance.run?.teardown();
}
}
await runs.remove(input.hash, input.ids);
});
const terminate = publicProcedure.input(z.string()).mutation(async ({ input, ctx }) => {
const { runtime } = ctx;
const { runner } = runtime;
const instance = runner.getInstance(input);
if (!instance || !instance.run) {
return;
}
await instance.run.teardown();
}); });
const runsRouter = router({ const runsRouter = router({
create, create,
find, find,
remove, remove,
prepareRemove,
terminate,
}); });
export { runsRouter }; export { runsRouter };

View File

@@ -1,5 +1,5 @@
import { EventEmitter } from 'eventemitter3'; import { EventEmitter } from 'eventemitter3';
import { RunInfo, run } from '@morten-olsen/mini-loader-runner'; import { run } from '@morten-olsen/mini-loader-runner';
import { Repos } from '../repos/repos.js'; import { Repos } from '../repos/repos.js';
import { LoggerEvent } from '../../../mini-loader/dist/esm/logger/logger.js'; import { LoggerEvent } from '../../../mini-loader/dist/esm/logger/logger.js';
import { ArtifactCreateEvent } from '../../../mini-loader/dist/esm/artifacts/artifacts.js'; import { ArtifactCreateEvent } from '../../../mini-loader/dist/esm/artifacts/artifacts.js';
@@ -20,17 +20,12 @@ type RunnerInstanceOptions = {
class RunnerInstance extends EventEmitter<RunnerInstanceEvents> { class RunnerInstance extends EventEmitter<RunnerInstanceEvents> {
#options: RunnerInstanceOptions; #options: RunnerInstanceOptions;
#run?: RunInfo;
constructor(options: RunnerInstanceOptions) { constructor(options: RunnerInstanceOptions) {
super(); super();
this.#options = options; this.#options = options;
} }
public get run() {
return this.#run;
}
#addLog = async (event: LoggerEvent['payload']) => { #addLog = async (event: LoggerEvent['payload']) => {
const { repos, id, loadId } = this.#options; const { repos, id, loadId } = this.#options;
const { logs } = repos; const { logs } = repos;
@@ -63,13 +58,11 @@ class RunnerInstance extends EventEmitter<RunnerInstanceEvents> {
const script = await readFile(scriptLocation, 'utf-8'); const script = await readFile(scriptLocation, 'utf-8');
const allSecrets = await secrets.getAll(); const allSecrets = await secrets.getAll();
await runs.started(id); await runs.started(id);
const current = await run({ const { promise, emitter } = await run({
script, script,
secrets: allSecrets, secrets: allSecrets,
input, input,
}); });
this.#run = current;
const { promise, emitter } = current;
emitter.on('message', (message) => { emitter.on('message', (message) => {
switch (message.type) { switch (message.type) {
case 'log': { case 'log': {
@@ -91,11 +84,9 @@ class RunnerInstance extends EventEmitter<RunnerInstanceEvents> {
} }
await runs.finished(id, { status: 'failed', error: errorMessage }); await runs.finished(id, { status: 'failed', error: errorMessage });
} finally { } finally {
this.#run = undefined;
this.emit('completed', { id }); this.emit('completed', { id });
} }
}; };
} }
export type { RunInfo };
export { RunnerInstance }; export { RunnerInstance };

View File

@@ -36,10 +36,6 @@ class Runner {
this.#instances.set(args.id, instance); this.#instances.set(args.id, instance);
await instance.start(); await instance.start();
}; };
public getInstance = (id: string) => {
return this.#instances.get(id);
};
} }
export { Runner }; export { Runner };

View File

@@ -3,16 +3,9 @@ import fastify from 'fastify';
import { RootRouter, rootRouter } from '../router/router.js'; import { RootRouter, rootRouter } from '../router/router.js';
import { createContext } from '../router/router.utils.js'; import { createContext } from '../router/router.utils.js';
import { Runtime } from '../runtime/runtime.js'; import { Runtime } from '../runtime/runtime.js';
import { gateway } from '../gateway/gateway.js';
const createServer = async (runtime: Runtime) => { const createServer = async (runtime: Runtime) => {
const server = fastify({ const server = fastify({});
maxParamLength: 10000,
bodyLimit: 30 * 1024 * 1024,
logger: {
level: 'warn',
},
});
server.get('/', async () => { server.get('/', async () => {
return { hello: 'world' }; return { hello: 'world' };
}); });
@@ -40,14 +33,6 @@ const createServer = async (runtime: Runtime) => {
}, },
} satisfies FastifyTRPCPluginOptions<RootRouter>['trpcOptions'], } satisfies FastifyTRPCPluginOptions<RootRouter>['trpcOptions'],
}); });
server.register(gateway, {
runtime,
});
server.addHook('onError', async (request, reply, error) => {
console.error(error);
});
await server.ready(); await server.ready();
return server; return server;

61
pnpm-lock.yaml generated
View File

@@ -36,9 +36,6 @@ importers:
packages/cli: packages/cli:
dependencies: dependencies:
'@morten-olsen/mini-loader-runner':
specifier: workspace:^
version: link:../runner
'@rollup/plugin-auto-install': '@rollup/plugin-auto-install':
specifier: ^3.0.5 specifier: ^3.0.5
version: 3.0.5(rollup@4.9.4) version: 3.0.5(rollup@4.9.4)
@@ -63,9 +60,6 @@ importers:
commander: commander:
specifier: ^11.1.0 specifier: ^11.1.0
version: 11.1.0 version: 11.1.0
dotenv:
specifier: ^16.3.1
version: 16.3.1
env-paths: env-paths:
specifier: ^3.0.0 specifier: ^3.0.0
version: 3.0.0 version: 3.0.0
@@ -91,6 +85,9 @@ importers:
'@morten-olsen/mini-loader-configs': '@morten-olsen/mini-loader-configs':
specifier: workspace:^ specifier: workspace:^
version: link:../configs version: link:../configs
'@morten-olsen/mini-loader-runner':
specifier: workspace:^
version: link:../runner
'@morten-olsen/mini-loader-server': '@morten-olsen/mini-loader-server':
specifier: workspace:^ specifier: workspace:^
version: link:../server version: link:../server
@@ -104,10 +101,6 @@ importers:
packages/configs: {} packages/configs: {}
packages/examples: packages/examples:
dependencies:
fastify:
specifier: ^4.25.2
version: 4.25.2
devDependencies: devDependencies:
'@morten-olsen/mini-loader': '@morten-olsen/mini-loader':
specifier: workspace:^ specifier: workspace:^
@@ -139,9 +132,6 @@ importers:
packages/runner: packages/runner:
dependencies: dependencies:
'@morten-olsen/mini-loader':
specifier: workspace:^
version: link:../mini-loader
eventemitter3: eventemitter3:
specifier: ^5.0.1 specifier: ^5.0.1
version: 5.0.1 version: 5.0.1
@@ -149,6 +139,9 @@ importers:
specifier: ^5.0.4 specifier: ^5.0.4
version: 5.0.4 version: 5.0.4
devDependencies: devDependencies:
'@morten-olsen/mini-loader':
specifier: workspace:^
version: link:../mini-loader
'@morten-olsen/mini-loader-configs': '@morten-olsen/mini-loader-configs':
specifier: workspace:^ specifier: workspace:^
version: link:../configs version: link:../configs
@@ -161,9 +154,6 @@ importers:
packages/server: packages/server:
dependencies: dependencies:
'@fastify/reply-from':
specifier: ^9.7.0
version: 9.7.0
'@trpc/client': '@trpc/client':
specifier: ^10.45.0 specifier: ^10.45.0
version: 10.45.0(@trpc/server@10.45.0) version: 10.45.0(@trpc/server@10.45.0)
@@ -486,11 +476,6 @@ packages:
fast-uri: 2.3.0 fast-uri: 2.3.0
dev: false dev: false
/@fastify/busboy@2.1.0:
resolution: {integrity: sha512-+KpH+QxZU7O4675t3mnkQKcZZg56u+K/Ct2K+N2AZYNVK8kyeo/bI18tI8aPm3tvNNRyTWfj6s5tnGNlcbQRsA==}
engines: {node: '>=14'}
dev: false
/@fastify/deepmerge@1.3.0: /@fastify/deepmerge@1.3.0:
resolution: {integrity: sha512-J8TOSBq3SoZbDhM9+R/u77hP93gz/rajSA+K2kGyijPpORPWUXHUpTaleoj+92As0S9uPRP7Oi8IqMf0u+ro6A==} resolution: {integrity: sha512-J8TOSBq3SoZbDhM9+R/u77hP93gz/rajSA+K2kGyijPpORPWUXHUpTaleoj+92As0S9uPRP7Oi8IqMf0u+ro6A==}
dev: false dev: false
@@ -505,19 +490,6 @@ packages:
fast-json-stringify: 5.10.0 fast-json-stringify: 5.10.0
dev: false dev: false
/@fastify/reply-from@9.7.0:
resolution: {integrity: sha512-/F1QBl3FGlTqStjmiuoLRDchVxP967TZh6FZPwQteWhdLsDec8mqSACE+cRzw6qHUj3v9hfdd7JNgmb++fyFhQ==}
dependencies:
'@fastify/error': 3.4.1
end-of-stream: 1.4.4
fast-content-type-parse: 1.1.0
fast-querystring: 1.1.2
fastify-plugin: 4.5.1
pump: 3.0.0
tiny-lru: 11.2.5
undici: 5.28.2
dev: false
/@gar/promisify@1.1.3: /@gar/promisify@1.1.3:
resolution: {integrity: sha512-k2Ty1JcVojjJFwrg/ThKi2ujJ7XNLYaFGNB/bWT9wGR+oSMJHMa5w+CUq6p/pVrKeNNgA7pCqEcjSnHVoqJQFw==} resolution: {integrity: sha512-k2Ty1JcVojjJFwrg/ThKi2ujJ7XNLYaFGNB/bWT9wGR+oSMJHMa5w+CUq6p/pVrKeNNgA7pCqEcjSnHVoqJQFw==}
requiresBuild: true requiresBuild: true
@@ -2192,11 +2164,6 @@ packages:
esutils: 2.0.3 esutils: 2.0.3
dev: true dev: true
/dotenv@16.3.1:
resolution: {integrity: sha512-IPzF4w4/Rd94bA9imS68tZBaYyBWSCE47V1RGuMrB94iyTOIEwRmVL2x/4An+6mETpLrKJ5hQkB8W4kFAadeIQ==}
engines: {node: '>=12'}
dev: false
/eastasianwidth@0.2.0: /eastasianwidth@0.2.0:
resolution: {integrity: sha512-I88TYZWc9XiYHRQ4/3c5rjjfgkjhLyW2luGIheGERbNQ6OY7yTybanSpDXZa8y7VUP9YmDcYa+eyq4ca7iLqWA==} resolution: {integrity: sha512-I88TYZWc9XiYHRQ4/3c5rjjfgkjhLyW2luGIheGERbNQ6OY7yTybanSpDXZa8y7VUP9YmDcYa+eyq4ca7iLqWA==}
dev: false dev: false
@@ -2716,10 +2683,6 @@ packages:
resolution: {integrity: sha512-eel5UKGn369gGEWOqBShmFJWfq/xSJvsgDzgLYC845GneayWvXBf0lJCBn5qTABfewy1ZDPoaR5OZCP+kssfuw==} resolution: {integrity: sha512-eel5UKGn369gGEWOqBShmFJWfq/xSJvsgDzgLYC845GneayWvXBf0lJCBn5qTABfewy1ZDPoaR5OZCP+kssfuw==}
dev: false dev: false
/fastify-plugin@4.5.1:
resolution: {integrity: sha512-stRHYGeuqpEZTL1Ef0Ovr2ltazUT9g844X5z/zEBFLG8RYlpDiOCIG+ATvYEp+/zmc7sN29mcIMp8gvYplYPIQ==}
dev: false
/fastify@4.25.2: /fastify@4.25.2:
resolution: {integrity: sha512-SywRouGleDHvRh054onj+lEZnbC1sBCLkR0UY3oyJwjD4BdZJUrxBqfkfCaqn74pVCwBaRHGuL3nEWeHbHzAfw==} resolution: {integrity: sha512-SywRouGleDHvRh054onj+lEZnbC1sBCLkR0UY3oyJwjD4BdZJUrxBqfkfCaqn74pVCwBaRHGuL3nEWeHbHzAfw==}
dependencies: dependencies:
@@ -5188,11 +5151,6 @@ packages:
engines: {node: '>=8'} engines: {node: '>=8'}
dev: false dev: false
/tiny-lru@11.2.5:
resolution: {integrity: sha512-JpqM0K33lG6iQGKiigcwuURAKZlq6rHXfrgeL4/I8/REoyJTGU+tEMszvT/oTRVHG2OiylhGDjqPp1jWMlr3bw==}
engines: {node: '>=12'}
dev: false
/tmp@0.0.33: /tmp@0.0.33:
resolution: {integrity: sha512-jRCJlojKnZ3addtTOjdIqoRuPEKBvNXcGYqzO6zWZX8KfKEpnGY5jfggJQ3EjKuu8D4bJRr0y+cYJFmYbImXGw==} resolution: {integrity: sha512-jRCJlojKnZ3addtTOjdIqoRuPEKBvNXcGYqzO6zWZX8KfKEpnGY5jfggJQ3EjKuu8D4bJRr0y+cYJFmYbImXGw==}
engines: {node: '>=0.6.0'} engines: {node: '>=0.6.0'}
@@ -5410,13 +5368,6 @@ packages:
/undici-types@5.26.5: /undici-types@5.26.5:
resolution: {integrity: sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA==} resolution: {integrity: sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA==}
/undici@5.28.2:
resolution: {integrity: sha512-wh1pHJHnUeQV5Xa8/kyQhO7WFa8M34l026L5P/+2TYiakvGy5Rdc8jWZVyG7ieht/0WgJLEd3kcU5gKx+6GC8w==}
engines: {node: '>=14.0'}
dependencies:
'@fastify/busboy': 2.1.0
dev: false
/unique-filename@1.1.1: /unique-filename@1.1.1:
resolution: {integrity: sha512-Vmp0jIp2ln35UTXuryvjzkjGdRyf9b2lTXuSYUiPmzRcl3FDtYqAwOnTJkAngD9SWhnoJzDbTKwaOrZ+STtxNQ==} resolution: {integrity: sha512-Vmp0jIp2ln35UTXuryvjzkjGdRyf9b2lTXuSYUiPmzRcl3FDtYqAwOnTJkAngD9SWhnoJzDbTKwaOrZ+STtxNQ==}
requiresBuild: true requiresBuild: true

View File

@@ -1,9 +1,7 @@
import { findWorkspacePackages } from '@pnpm/find-workspace-packages'; import { findWorkspacePackages } from '@pnpm/find-workspace-packages';
import { readFile, writeFile } from 'fs/promises'; import { writeFile } from 'fs/promises';
import { join } from 'path'; import { join } from 'path';
const sharedData = JSON.parse(await readFile(join(process.cwd(), 'scripts/shared-data.json')));
const version = process.argv[2]; const version = process.argv[2];
if (!version) { if (!version) {
throw new Error('Version is required'); throw new Error('Version is required');
@@ -13,9 +11,6 @@ const packages = await findWorkspacePackages(process.cwd());
for (const { manifest, dir } of packages) { for (const { manifest, dir } of packages) {
console.log(dir, version); console.log(dir, version);
for (let [key, value] of Object.entries(sharedData || {})) {
manifest[key] = value;
}
manifest.version = version; manifest.version = version;
await writeFile(join(dir, 'package.json'), JSON.stringify(manifest, null, 2)); await writeFile(join(dir, 'package.json'), JSON.stringify(manifest, null, 2));
} }

View File

@@ -1,8 +0,0 @@
{
"license": "GPL-3.0",
"homepage": "https://github.com/morten-olsen/mini-loader",
"repository": {
"type": "git",
"url": "https://github.com/morten-olsen/mini-loader"
}
}

View File

@@ -1,15 +1,15 @@
{ {
"include": [], "include": [],
"references": [ "references": [
{
"path": "./packages/mini-loader/tsconfig.json"
},
{ {
"path": "./packages/runner/tsconfig.json" "path": "./packages/runner/tsconfig.json"
}, },
{ {
"path": "./packages/server/tsconfig.json" "path": "./packages/server/tsconfig.json"
}, },
{
"path": "./packages/mini-loader/tsconfig.json"
},
{ {
"path": "./packages/cli/tsconfig.json" "path": "./packages/cli/tsconfig.json"
}, },