1053 Commits

Author SHA1 Message Date
Jason Song
7497155d14 feat: update step state 2022-11-25 17:48:45 +08:00
Jason Song
6df53243aa feat: create task steps 2022-11-25 17:48:45 +08:00
fuxiaohei
9648482ed6 feat(bots): move runner status update function to context 2022-11-25 17:48:45 +08:00
fuxiaohei
1a78fd3494 feat(runner): update runner last_online and status when FetchTask 2022-11-25 17:48:44 +08:00
Jason Song
32a5797608 chore: debug runner log reporting 2022-11-25 17:48:44 +08:00
Jason Song
79966763c6 feat: distribute tasks to runner 2022-11-25 17:48:44 +08:00
Bo-Yi.Wu
6f18f76577 chore(runner): update runner status using proto define
Signed-off-by: Bo-Yi.Wu <appleboy.tw@gmail.com>
2022-11-25 17:48:44 +08:00
Jason Song
530faf7227 feat: assign task to runner 2022-11-25 17:48:44 +08:00
Bo-Yi.Wu
4233090738 chore(runner): get token from header
Signed-off-by: Bo-Yi.Wu <appleboy.tw@gmail.com>
2022-11-25 17:48:44 +08:00
Bo-Yi.Wu
f79bc4f605 chore(runner): vaildate the repo URL
Signed-off-by: Bo-Yi.Wu <appleboy.tw@gmail.com>
2022-11-25 17:48:44 +08:00
Bo-Yi.Wu
ade88b3a7b chore(runner): register new runner
Signed-off-by: Bo-Yi.Wu <appleboy.tw@gmail.com>
2022-11-25 17:48:44 +08:00
Jason Song
e21c07cc55 feat: parse runner from ctx 2022-11-25 17:48:44 +08:00
Bo-Yi.Wu
4dd135580f chore(runner): update to latest proto format.
Signed-off-by: Bo-Yi.Wu <appleboy.tw@gmail.com>
2022-11-25 17:48:44 +08:00
Bo-Yi.Wu
fb738c4192 chore(runner): support fetch detail stage data
Signed-off-by: Bo-Yi.Wu <appleboy.tw@gmail.com>
2022-11-25 17:48:44 +08:00
Bo-Yi.Wu
8255915350 chore: move core package to top folder.
Signed-off-by: Bo-Yi.Wu <appleboy.tw@gmail.com>
2022-11-25 17:48:44 +08:00
Bo-Yi.Wu
1148903d79 refactor: move build status to core package 2022-11-25 17:48:44 +08:00
Bo-Yi.Wu
ea0cf8515e chore(gRPC): handle requesut for stage data
Signed-off-by: Bo-Yi.Wu <appleboy.tw@gmail.com>
2022-11-25 17:48:44 +08:00
Bo-Yi.Wu
eac6425b2f chore(gRPC): support update stage and step method.
Signed-off-by: Bo-Yi.Wu <appleboy.tw@gmail.com>
2022-11-25 17:48:44 +08:00
Bo-Yi.Wu
68d416bd1b chore(gRPC): initial request method
Signed-off-by: Bo-Yi.Wu <appleboy.tw@gmail.com>
2022-11-25 17:48:44 +08:00
Bo-Yi.Wu
e07d416366 chore(model): create build, stage, step and logs table
Signed-off-by: Bo-Yi.Wu <appleboy.tw@gmail.com>
2022-11-25 17:48:44 +08:00
Bo-Yi.Wu
e05f224d19 chore(runner): support register
Signed-off-by: Bo-Yi.Wu <appleboy.tw@gmail.com>
2022-11-25 17:48:44 +08:00
Bo-Yi.Wu
c10414a018 test(ping): add testing for gRPC 2022-11-25 17:48:44 +08:00
Bo-Yi.Wu
e4e41a8065 chore(bots): re-structure grpc folder.
Signed-off-by: Bo-Yi.Wu <appleboy.tw@gmail.com>
2022-11-25 17:48:44 +08:00
Bo-Yi.Wu
2cba8fafd8 chore(grpc): add health check and grpc v1, alpha version
Signed-off-by: Bo-Yi.Wu <appleboy.tw@gmail.com>
2022-11-25 17:48:44 +08:00
GiteaBot
257ba77e8d chore(proto): update import path
Signed-off-by: GiteaBot <teabot@gitea.io>
2022-11-25 17:48:44 +08:00
Bo-Yi Wu
ef3a74a1ba chore(proto): Add ping service
Signed-off-by: Bo-Yi Wu <appleboy.tw@gmail.com>
2022-11-25 17:48:44 +08:00
Bo-Yi Wu
0e79fc556a chore(router): Add grpc and grep-web api
Using connect-go package

https://github.com/bufbuild/connect-go

Signed-off-by: Bo-Yi Wu <appleboy.tw@gmail.com>
2022-11-25 17:48:44 +08:00
Lunny Xiao
931d8c2c21 add runners management ui 2022-11-25 17:48:43 +08:00
Lunny Xiao
2c4f6fd42f add most tables 2022-11-25 17:48:43 +08:00
Lunny Xiao
5a479bb034 Add builds UI 2022-11-25 17:48:43 +08:00
Lunny Xiao
7732392a96 Add bots 2022-11-25 17:48:33 +08:00
KN4CK3R
a1ae83f36e
Workaround for container registry push/pull errors (#21862)
This PR addresses #19586

I added a mutex to the upload version creation which will prevent the
push errors when two requests try to create these database entries. I'm
not sure if this should be the final solution for this problem.

I added a workaround to allow a reupload of missing blobs. Normally a
reupload is skipped because the database knows the blob is already
present. The workaround checks if the blob exists on the file system.
This should not be needed anymore with the above fix so I marked this
code to be removed with Gitea v1.20.

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-11-25 13:47:46 +08:00
KN4CK3R
fc7a2d5a95
Add support for HEAD requests in Maven registry (#21834)
Related #18543

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-11-24 16:25:13 +02:00
Jason Song
e4eaa68a2b
Replace yaml.v2 with yaml.v3 (#21832)
I don't see why we have to use two versions of yaml. The difference
between the two versions has nothing to do with our usage.
2022-11-21 16:36:59 +08:00
KN4CK3R
044c754ea5
Add context.Context to more methods (#21546)
This PR adds a context parameter to a bunch of methods. Some helper
`xxxCtx()` methods got replaced with the normal name now.

Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-11-19 16:12:33 +08:00
KN4CK3R
43ab9324c5
Fix setting HTTP headers after write (#21833)
The headers can't be modified after it was send to the client.
2022-11-18 01:55:15 +08:00
Lunny Xiao
34283a74e8
Allow detect whether it's in a database transaction for a context.Context (#21756)
Fix #19513

This PR introduce a new db method `InTransaction(context.Context)`,
and also builtin check on `db.TxContext` and `db.WithTx`.
There is also a new method `db.AutoTx` has been introduced but could be used by other PRs.

`WithTx` will always open a new transaction, if a transaction exist in context, return an error.
`AutoTx` will try to open a new transaction if no transaction exist in context.
That means it will always enter a transaction if there is no error.

Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: 6543 <6543@obermui.de>
2022-11-12 21:18:50 +01:00
zeripath
a0a425a13b
Add some documentation to packages (#21648)
In #21637 it was mentioned that the purpose of the API routes for the
packages is unclear. This PR adds some documentation.

Fix #21637

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
2022-11-12 18:59:15 +00:00
Jason Song
bf2078640f
Load GitRepo in API before deleting issue (#21720)
Fix #20921.

The `ctx.Repo.GitRepo` has been used in deleting issues when the issue
is a PR.

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
2022-11-12 08:32:04 +08:00
Drew Noel
2cbea23d70
Add configuration for CORS allowed headers (#21747)
This PR enhances the CORS middleware usage by allowing for the headers
to be configured in `app.ini`.

Fixes #21746

Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-11-11 14:39:27 +08:00
KN4CK3R
20674dd05d
Add package registry quota limits (#21584)
Related #20471

This PR adds global quota limits for the package registry. Settings for
individual users/orgs can be added in a seperate PR using the settings
table.

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-11-09 14:34:27 +08:00
Wayne Starr
8c1d9885e5
Remove semver compatible flag and change pypi to an array of test cases (#21708)
This addresses #21707 and adds a second package test case for a
non-semver compatible version (this might be overkill though since you
could also edit the old package version to have an epoch in front and
see the error, this just seemed more flexible for the future).

Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
2022-11-08 09:41:39 +08:00
KN4CK3R
fd89c062bd
Allow local package identifiers for PyPI packages (#21690)
Fixes #21683

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-11-07 21:35:04 +02:00
oliverpool
b6e81357bd
Add Webhook authorization header (#20926)
_This is a different approach to #20267, I took the liberty of adapting
some parts, see below_

## Context

In some cases, a weebhook endpoint requires some kind of authentication.
The usual way is by sending a static `Authorization` header, with a
given token. For instance:

- Matrix expects a `Bearer <token>` (already implemented, by storing the
header cleartext in the metadata - which is buggy on retry #19872)
- TeamCity #18667
- Gitea instances #20267
- SourceHut https://man.sr.ht/graphql.md#authentication-strategies (this
is my actual personal need :)

## Proposed solution

Add a dedicated encrypt column to the webhook table (instead of storing
it as meta as proposed in #20267), so that it gets available for all
present and future hook types (especially the custom ones #19307).

This would also solve the buggy matrix retry #19872.

As a first step, I would recommend focusing on the backend logic and
improve the frontend at a later stage. For now the UI is a simple
`Authorization` field (which could be later customized with `Bearer` and
`Basic` switches):


![2022-08-23-142911](https://user-images.githubusercontent.com/3864879/186162483-5b721504-eef5-4932-812e-eb96a68494cc.png)

The header name is hard-coded, since I couldn't fine any usecase
justifying otherwise.

## Questions

- What do you think of this approach? @justusbunsi @Gusted @silverwind 
- ~~How are the migrations generated? Do I have to manually create a new
file, or is there a command for that?~~
- ~~I started adding it to the API: should I complete it or should I
drop it? (I don't know how much the API is actually used)~~

## Done as well:

- add a migration for the existing matrix webhooks and remove the
`Authorization` logic there


_Closes #19872_

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Gusted <williamzijl7@hotmail.com>
Co-authored-by: delvh <dev.lh@web.de>
2022-11-03 20:23:20 +02:00
kolaente
085f717529
feat: notify doers of a merge when automerging (#21553)
I found myself wondering whether a PR I scheduled for automerge was
actually merged. It was, but I didn't receive a mail notification for it
- that makes sense considering I am the doer and usually don't want to
receive such notifications. But ideally I want to receive a notification
when a PR was merged because I scheduled it for automerge.

This PR implements exactly that.

The implementation works, but I wonder if there's a way to avoid passing
the "This PR was automerged" state down so much. I tried solving this
via the database (checking if there's an automerge scheduled for this PR
when sending the notification) but that did not work reliably, probably
because sending the notification happens async and the entry might have
already been deleted. My implementation might be the most
straightforward but maybe not the most elegant.

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-11-03 23:49:00 +08:00
qwerty287
8430f738e2
Fix 500 on PR files API (#21602)
Fixes an 500 error/panic if using the changed PR files API with pages
that should return empty lists because there are no items anymore.
`start-end` is then < 0 which ends in panic.

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: delvh <dev.lh@web.de>
2022-10-26 23:46:11 +08:00
delvh
0ebb45cfe7
Replace all instances of fmt.Errorf(%v) with fmt.Errorf(%w) (#21551)
Found using
`find . -type f -name '*.go' -print -exec vim {} -c
':%s/fmt\.Errorf(\(.*\)%v\(.*\)err/fmt.Errorf(\1%w\2err/g' -c ':wq' \;`

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-10-24 20:29:17 +01:00
KN4CK3R
7c11a73833
Fix package access for admins and inactive users (#21580)
I noticed an admin is not allowed to upload packages for other users
because `ctx.IsSigned` was not set.
I added a check for `user.IsActive` and `user.ProhibitLogin` too because
both was not checked. Tests enforce this now.

Co-authored-by: Lauris BH <lauris@nix.lv>
2022-10-24 22:23:25 +03:00
Wayne Starr
49a4464160
Allow for resolution of NPM registry paths that match upstream (#21568)
This PR fixes issue #21567 allowing for package tarball URLs to match
the upstream registry (and GitLab/JFrog Artifactory URLs). It uses a
regex to parse the filename (which contains the NPM version) and does a
fuzzy search to pull it out. The regex was built/expanded from
http://json.schemastore.org/package,
https://github.com/Masterminds/semver, and
https://docs.npmjs.com/cli/v6/using-npm/semver and is testable here:
https://regex101.com/r/OydBJq/5

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-10-24 21:50:22 +08:00
M Hickford
191a74d622
Record OAuth client type at registration (#21316)
The OAuth spec [defines two types of
client](https://datatracker.ietf.org/doc/html/rfc6749#section-2.1),
confidential and public. Previously Gitea assumed all clients to be
confidential.

> OAuth defines two client types, based on their ability to authenticate
securely with the authorization server (i.e., ability to
>   maintain the confidentiality of their client credentials):
>
>   confidential
> Clients capable of maintaining the confidentiality of their
credentials (e.g., client implemented on a secure server with
> restricted access to the client credentials), or capable of secure
client authentication using other means.
>
>   **public
> Clients incapable of maintaining the confidentiality of their
credentials (e.g., clients executing on the device used by the resource
owner, such as an installed native application or a web browser-based
application), and incapable of secure client authentication via any
other means.**
>
> The client type designation is based on the authorization server's
definition of secure authentication and its acceptable exposure levels
of client credentials. The authorization server SHOULD NOT make
assumptions about the client type.

 https://datatracker.ietf.org/doc/html/rfc8252#section-8.4

> Authorization servers MUST record the client type in the client
registration details in order to identify and process requests
accordingly.

Require PKCE for public clients:
https://datatracker.ietf.org/doc/html/rfc8252#section-8.1

> Authorization servers SHOULD reject authorization requests from native
apps that don't use PKCE by returning an error message

Fixes #21299

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-10-24 15:59:24 +08:00