Compare commits

...

41 Commits

Author SHA1 Message Date
argo-bot
fe80bdcfdc Bump version to 2.4.4 2022-07-07 06:57:43 +00:00
argo-bot
94c09dff59 Bump version to 2.4.4 2022-07-07 06:57:37 +00:00
jannfis
3d9e4d439e test: Remove circular symlinks from testdata (#9886)
* test: Remove circular symlinks from testdata

Signed-off-by: jannfis <jann@mistrust.net>

* Another test case

Signed-off-by: jannfis <jann@mistrust.net>

* Use defer for changing back to original workdir

Signed-off-by: jannfis <jann@mistrust.net>

* Abort the test on error in defer

Signed-off-by: jannfis <jann@mistrust.net>
2022-07-06 19:18:51 +00:00
noah
4a955e25a0 docs: custom secret must be labeled (#9835)
As far as I can tell, this isn't explicitly documented anywhere (some docs mentioned this label in reference to ConfigMaps) I only figure it out by looking at the code.
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
2022-07-01 15:55:34 -04:00
noah
4f99f251bf fix: missing path segments for git file generator (#9839)
* fix: missing path segments for git file generator

given the path:

  /one/two/file.yaml

This change adds the params:

path[1]=two
path.filename=file.yaml
path.filenameNormalized

The use case is for symmetry with other generators (e.g., merge with a directory generator using the key path[1]). As no existing parameter values are changed it should be non-breaking.

Signed-off-by: Noah Perks Sloan <noah@hackedu.io>

* fix: expose all path elements for directory generator

This makes it consistent with the files generator and removes un-intuitive behavior without any breaking change.

docs: clarify - can use either baseName or nth path element
docs: use "directory", like the name of the generator, where "folder" was mentioned
Signed-off-by: Noah Perks Sloan <noah@hackedu.io>
2022-07-01 15:54:15 -04:00
Alexander Matyushentsev
fd418cec0d fix: make sure api server informer does not stop after setting change (#9842)
Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>
2022-06-30 15:30:33 -07:00
Xiao Yang
dc291a629f fix: support resource logs and exec (#9833)
Signed-off-by: Xiao Yang <muma.378@163.com>
2022-06-30 09:07:00 -04:00
Paulo Coelho
e23a1c6026 docs: update archlinux install with official package (#9718)
Signed-off-by: Paulo Coelho <prscoelho@protonmail.com>
2022-06-29 16:38:21 -04:00
Chris Reilly
aee88c6452 docs: explain rightmost git generator path parameter behavior (#9799)
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
2022-06-29 16:31:15 -04:00
KevinSnyderCodes
3f111cc640 git: prune any deleted refs before fetching (#9504)
* git: prune any deleted refers before fetching

This commit modifies `nativeGitClient.Fetch()` to call `git remote prune origin` before fetching refs.

In some cases, an old branch may exist that conflicts with the name of a new branch. The old branch will have been deleted from `origin` but still exist locally in the `argocd-repo-server`.

Example: an old branch `feature/foo` conflicts with a new branch `feature/foo/bar`

In these cases, syncing an application results in the error:

```
rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force` failed exit status 1: error: cannot lock ref 'refs/remotes/origin/feature/foo/bar': 'refs/remotes/origin/feature/foo' exists; cannot create 'refs/remotes/origin/feature/foo/bar' From https://github.com/org/repo ! [new branch] feature/foo/bar -> origin/feature/foo/bar (unable to update local ref) error: some local refs could not be updated; try running 'git remote prune origin' to remove any old, conflicting branches
```

Adding `git remote prune origin` before fetching, as recommended by the error message, should fix this issue.

The current workaround is to restart the `argocd-repo-server` which should flush the local repository folder. This works when Argo CD is installed using the Helm chart.

Signed-off-by: Kevin Snyder <kevin.snyder.codes@gmail.com>

* fix: added extra protection to syncing app with replace (#9187)

* fix: added extra protection to syncing app with replace

Signed-off-by: ciiay <yicai@redhat.com>

* Code clean up

Signed-off-by: ciiay <yicai@redhat.com>

* Updated logic for isAppOfAppsPattern

Signed-off-by: ciiay <yicai@redhat.com>

* Updated text strings as per comment

Signed-off-by: ciiay <yicai@redhat.com>

* Fixed lint issue

Signed-off-by: ciiay <yicai@redhat.com>
Signed-off-by: Kevin Snyder <kevin.snyder.codes@gmail.com>

* chore: Simplified GetRepoHTTPClient function (#9396)

* chore: Simplified GetRepoHTTPClient function

Signed-off-by: ls0f <lovedboy.tk@qq.com>

* simplified code and improve unit test coverage

Signed-off-by: ls0f <lovedboy.tk@qq.com>
Signed-off-by: Kevin Snyder <kevin.snyder.codes@gmail.com>

* Only prune if fetch error message indicates that it is worthwhile, add unit tests

Confirmed that `Test_nativeGitClient_Fetch_Prune` fails without the bug fix, succeeds with it.

Signed-off-by: Kevin Snyder <kevin.snyder.codes@gmail.com>

* fix: avoid k8s call before authorization for terminal endpoint (#9434)

* fix: avoid k8s API call before authorization in k8s endpoint

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* check for bad project

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* lint

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* more logging

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* handle 404, return 500 instead of 400 for other errors

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* use user input

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* refactor validation

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* fix tests

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* fixes, tests

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
Signed-off-by: Kevin Snyder <kevin.snyder.codes@gmail.com>

* Match against "try running 'git remote prune origin'"

Signed-off-by: Kevin Snyder <kevin.snyder.codes@gmail.com>

Co-authored-by: Yi Cai <yicai@redhat.com>
Co-authored-by: ls0f <lovedboy.tk@qq.com>
Co-authored-by: Michael Crenshaw <michael@crenshaw.dev>
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
2022-06-29 16:28:51 -04:00
Jake
596038fc0f fix: configurable CMP tar exclusions (#9675) (#9789)
* feat: configurable CMP tar exclusions

Signed-off-by: notfromstatefarm <86763948+notfromstatefarm@users.noreply.github.com>

* rename cmp to plugin

Signed-off-by: notfromstatefarm <86763948+notfromstatefarm@users.noreply.github.com>

* add test for tar stream exclusions

Signed-off-by: notfromstatefarm <86763948+notfromstatefarm@users.noreply.github.com>

* fix tests

Signed-off-by: notfromstatefarm <86763948+notfromstatefarm@users.noreply.github.com>

* update cmp guide

Signed-off-by: notfromstatefarm <86763948+notfromstatefarm@users.noreply.github.com>

* fully parameterize

Signed-off-by: notfromstatefarm <86763948+notfromstatefarm@users.noreply.github.com>
2022-06-29 16:27:41 -04:00
argo-bot
471685feae Bump version to 2.4.3 2022-06-27 20:53:14 +00:00
argo-bot
42e1f29117 Bump version to 2.4.3 2022-06-27 20:53:07 +00:00
Michael Crenshaw
c089f9a5e5 fix: respect OIDC providers' supported token signing algorithms (#9433) (#9761)
* fix: respect OIDC providers' supported token signing algorithms (#9433)

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* go mod tidy

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
2022-06-27 16:32:35 -04:00
TomHellier
1dc5e6aaf9 fix: websockets for terminal not working on subPath (#9795)
Signed-off-by: Tom Hellier <me@tomhellier.com>
2022-06-27 13:26:11 -07:00
Alexander Matyushentsev
82726fcbcf fix: avoid closing and re-opening port of api server settings change (#9778)
Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>
2022-06-24 10:17:20 -07:00
Nicolas Fillot
e715e085aa [ArgoCD] Fixing webhook typo in case of error in GetManifests (#9671)
Signed-off-by: Nicolas Fillot <nfillot@weborama.com>

Co-authored-by: Nicolas Fillot <nfillot@weborama.com>
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
2022-06-23 09:27:15 -04:00
Michael Crenshaw
7db5ab71b0 fix: overrides should not appear in the manifest cache key (#9601)
* fix: overrides should not appear in the manifest cache key

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* fix Helm regression

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* fix test

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* fix test again

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
2022-06-22 09:24:29 -04:00
argo-bot
c6d0c8baaa Bump version to 2.4.2 2022-06-21 20:32:13 +00:00
argo-bot
2115bf7746 Bump version to 2.4.2 2022-06-21 20:32:08 +00:00
Yaakov Selkowitz
0a2520f511 test/remote: Allow override of base image (#9734)
ubuntu:22.04 bumped the ppc64el baseline to POWER9, rendering this
unusable on existing POWER8 systems.  This allows customization similar
to the top-level Dockerfile.

Signed-off-by: Yaakov Selkowitz <yselkowi@redhat.com>
2022-06-21 15:24:13 -04:00
Soumya Ghosh Dastidar
211e9f6127 fix: updated config file permission requirements for windows (#9732)
* fix: reduced config file permission restriction on windows

Signed-off-by: Soumya Ghosh Dastidar <gdsoumya@gmail.com>

* fix: updated localconfig tests to check error

Signed-off-by: Soumya Ghosh Dastidar <gdsoumya@gmail.com>
2022-06-21 15:09:48 -04:00
hritikch24
792e278080 Update sync-options.md (#9687)
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
2022-06-21 14:49:10 -04:00
Michael Crenshaw
93bf321002 fix: project filter (#9651) (#9709)
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
2022-06-21 14:39:34 -04:00
Joren Zandstra
48c2e0bf21 fix broken symlink in Dockerfile (#9674)
Signed-off-by: Joren Zandstra <jorenzandstra@yahoo.com>
2022-06-21 14:39:26 -04:00
Soumya Ghosh Dastidar
b4b893ca1c fix: updated baseHRefRegex to perform lazy match (#9724)
* fix: updated baseHRefRegex to perform lazy match

Signed-off-by: Soumya Ghosh Dastidar <gdsoumya@gmail.com>

* feat: added unit tests for replaceBaseHRef

Signed-off-by: Soumya Ghosh Dastidar <gdsoumya@gmail.com>
2022-06-21 10:39:12 -07:00
argo-bot
52e6025f8b Bump version to 2.4.1 2022-06-21 16:27:47 +00:00
argo-bot
b121b89c81 Bump version to 2.4.1 2022-06-21 16:27:40 +00:00
Michael Crenshaw
12149c0710 chore: fix docs gen
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
2022-06-21 10:34:03 -04:00
Michael Crenshaw
9d67469428 Merge pull request from GHSA-jhqp-vf4w-rpwq
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

defer instead of multiple close calls

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

oops

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

don't count jsonnet against max

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

fix codegen

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

add caveat about 300x ratio

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

fix versions

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

fix tests/lint

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
2022-06-21 09:40:36 -04:00
Michael Crenshaw
04c3053964 Merge pull request from GHSA-q4w5-4gq2-98vm
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

fix tests/lint

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
2022-06-21 09:39:56 -04:00
Michael Crenshaw
17f7f4f462 Merge pull request from GHSA-2m7h-86qq-fp4v
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

fix references

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

use long enough state param for oauth2

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

typo

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

more entropy

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

fix tests

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

fix tests/lint

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
2022-06-21 09:39:01 -04:00
Michael Crenshaw
8bc3ef690d Merge pull request from GHSA-h4w9-6x78-8vrj
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

fix tests/lint

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
2022-06-21 09:36:38 -04:00
Michael Crenshaw
8fe9a58c21 test: directory app manifest generation (#9503)
* test: directory app manifest generation

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>

* git doesn't support empty dirs

Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
2022-06-14 16:12:12 -04:00
Leonardo Luz Almeida
269c61a9c8 chore: Implement tests to validate aws auth retry (#9627)
Signed-off-by: Leonardo Luz Almeida <leonardo_almeida@intuit.com>
2022-06-14 10:11:38 -04:00
Leonardo Luz Almeida
9a08c123f9 chore: Implement a retry in aws auth command (#9618)
Signed-off-by: Leonardo Luz Almeida <leonardo_almeida@intuit.com>
2022-06-14 10:10:43 -04:00
jannfis
971f0f1ff1 test: Remove temp directories from repo server tests (#9501)
Signed-off-by: jannfis <jann@mistrust.net>
2022-06-13 15:42:29 -04:00
jannfis
1ae2f97b05 test: Make context tests idempodent (#9502)
Signed-off-by: jannfis <jann@mistrust.net>
2022-06-13 15:41:55 -04:00
Michael Crenshaw
bb26cc207b test: fix plugin var test for OSX (#9590)
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
2022-06-13 15:37:00 -04:00
Christian Hernandez
77d3dcdc62 Update Generators-Git.md (#9632)
Document how to deploy from the root of the git repository

Signed-off-by: Christian Hernandez <christian@codefresh.io>
2022-06-13 10:57:28 -04:00
Hari Sekhon
f414a8e985 docs: added environment variables documentation (#8680)
* added environment variables page

Signed-off-by: Hari Sekhon <harisekhon@gmail.com>

* added Environment Variables page to User Guide

Signed-off-by: Hari Sekhon <harisekhon@gmail.com>

* reformatted table cells as multiline for clarity

Signed-off-by: Hari Sekhon <harisekhon@gmail.com>

* fixed typo

Co-authored-by: Michael Crenshaw <michael@crenshaw.dev>
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
2022-06-13 10:38:39 -04:00
110 changed files with 2220 additions and 497 deletions

View File

@@ -63,7 +63,7 @@ RUN ln -s /usr/local/bin/entrypoint.sh /usr/local/bin/uid_entrypoint.sh
# support for mounting configuration from a configmap
WORKDIR /app/config/ssh
RUN touch ssh_known_hosts && \
ln -s ssh_known_hosts /etc/ssh/ssh_known_hosts
ln -s /app/config/ssh/ssh_known_hosts /etc/ssh/ssh_known_hosts
WORKDIR /app/config
RUN mkdir -p tls && \

View File

@@ -37,6 +37,7 @@ Currently, the following organizations are **officially** using Argo CD:
1. [Chargetrip](https://chargetrip.com)
1. [Chime](https://www.chime.com)
1. [Cisco ET&I](https://eti.cisco.com/)
1. [Cobalt](https://www.cobalt.io/)
1. [Codefresh](https://www.codefresh.io/)
1. [Codility](https://www.codility.com/)
1. [Commonbond](https://commonbond.co/)

View File

@@ -1 +1 @@
2.4.0
2.4.4

View File

@@ -162,8 +162,10 @@ func (g *GitGenerator) generateParamsFromGitFile(filePath string, fileContent []
}
params["path"] = path.Dir(filePath)
params["path.basename"] = path.Base(params["path"])
params["path.filename"] = path.Base(filePath)
params["path.basenameNormalized"] = sanitizeName(path.Base(params["path"]))
for k, v := range strings.Split(strings.TrimSuffix(params["path"], params["path.basename"]), "/") {
params["path.filenameNormalized"] = sanitizeName(path.Base(params["path.filename"]))
for k, v := range strings.Split(params["path"], "/") {
if len(v) > 0 {
params["path["+strconv.Itoa(k)+"]"] = v
}
@@ -213,7 +215,7 @@ func (g *GitGenerator) generateParamsFromApps(requestedApps []string, _ *argopro
params["path"] = a
params["path.basename"] = path.Base(a)
params["path.basenameNormalized"] = sanitizeName(path.Base(a))
for k, v := range strings.Split(strings.TrimSuffix(params["path"], params["path.basename"]), "/") {
for k, v := range strings.Split(params["path"], "/") {
if len(v) > 0 {
params["path["+strconv.Itoa(k)+"]"] = v
}

View File

@@ -47,6 +47,28 @@ func (a argoCDServiceMock) GetDirectories(ctx context.Context, repoURL string, r
return args.Get(0).([]string), args.Error(1)
}
func Test_generateParamsFromGitFile(t *testing.T) {
params, err := (*GitGenerator)(nil).generateParamsFromGitFile("path/dir/file_name.yaml", []byte(`
foo:
bar: baz
`))
if err != nil {
t.Fatal(err)
}
assert.Equal(t, []map[string]string{
{
"foo.bar": "baz",
"path": "path/dir",
"path.basename": "dir",
"path.filename": "file_name.yaml",
"path.basenameNormalized": "dir",
"path.filenameNormalized": "file-name.yaml",
"path[0]": "path",
"path[1]": "dir",
},
}, params)
}
func TestGitGenerateParamsFromDirectories(t *testing.T) {
cases := []struct {
@@ -68,9 +90,9 @@ func TestGitGenerateParamsFromDirectories(t *testing.T) {
},
repoError: nil,
expected: []map[string]string{
{"path": "app1", "path.basename": "app1", "path.basenameNormalized": "app1"},
{"path": "app2", "path.basename": "app2", "path.basenameNormalized": "app2"},
{"path": "app_3", "path.basename": "app_3", "path.basenameNormalized": "app-3"},
{"path": "app1", "path.basename": "app1", "path.basenameNormalized": "app1", "path[0]": "app1"},
{"path": "app2", "path.basename": "app2", "path.basenameNormalized": "app2", "path[0]": "app2"},
{"path": "app_3", "path.basename": "app_3", "path.basenameNormalized": "app-3", "path[0]": "app_3"},
},
expectedError: nil,
},
@@ -85,8 +107,8 @@ func TestGitGenerateParamsFromDirectories(t *testing.T) {
},
repoError: nil,
expected: []map[string]string{
{"path": "p1/app2", "path.basename": "app2", "path[0]": "p1", "path.basenameNormalized": "app2"},
{"path": "p1/p2/app3", "path.basename": "app3", "path[0]": "p1", "path[1]": "p2", "path.basenameNormalized": "app3"},
{"path": "p1/app2", "path.basename": "app2", "path[0]": "p1", "path[1]": "app2", "path.basenameNormalized": "app2"},
{"path": "p1/p2/app3", "path.basename": "app3", "path[0]": "p1", "path[1]": "p2", "path[2]": "app3", "path.basenameNormalized": "app3"},
},
expectedError: nil,
},
@@ -102,9 +124,9 @@ func TestGitGenerateParamsFromDirectories(t *testing.T) {
},
repoError: nil,
expected: []map[string]string{
{"path": "app1", "path.basename": "app1", "path.basenameNormalized": "app1"},
{"path": "app2", "path.basename": "app2", "path.basenameNormalized": "app2"},
{"path": "p2/app3", "path.basename": "app3", "path[0]": "p2", "path.basenameNormalized": "app3"},
{"path": "app1", "path.basename": "app1", "path[0]": "app1", "path.basenameNormalized": "app1"},
{"path": "app2", "path.basename": "app2", "path[0]": "app2", "path.basenameNormalized": "app2"},
{"path": "p2/app3", "path.basename": "app3", "path[0]": "p2", "path[1]": "app3", "path.basenameNormalized": "app3"},
},
expectedError: nil,
},
@@ -120,9 +142,9 @@ func TestGitGenerateParamsFromDirectories(t *testing.T) {
},
repoError: nil,
expected: []map[string]string{
{"path": "app1", "path.basename": "app1", "path.basenameNormalized": "app1"},
{"path": "app2", "path.basename": "app2", "path.basenameNormalized": "app2"},
{"path": "p2/app3", "path.basename": "app3", "path[0]": "p2", "path.basenameNormalized": "app3"},
{"path": "app1", "path.basename": "app1", "path[0]": "app1", "path.basenameNormalized": "app1"},
{"path": "app2", "path.basename": "app2", "path[0]": "app2", "path.basenameNormalized": "app2"},
{"path": "p2/app3", "path.basename": "app3", "path[0]": "p2", "path[1]": "app3", "path.basenameNormalized": "app3"},
},
expectedError: nil,
},
@@ -238,7 +260,10 @@ func TestGitGenerateParamsFromFiles(t *testing.T) {
"path": "cluster-config/production",
"path.basename": "production",
"path[0]": "cluster-config",
"path[1]": "production",
"path.basenameNormalized": "production",
"path.filename": "config.json",
"path.filenameNormalized": "config.json",
},
{
"cluster.owner": "foo.bar@example.com",
@@ -247,7 +272,10 @@ func TestGitGenerateParamsFromFiles(t *testing.T) {
"path": "cluster-config/staging",
"path.basename": "staging",
"path[0]": "cluster-config",
"path[1]": "staging",
"path.basenameNormalized": "staging",
"path.filename": "config.json",
"path.filenameNormalized": "config.json",
},
},
expectedError: nil,
@@ -305,7 +333,10 @@ func TestGitGenerateParamsFromFiles(t *testing.T) {
"path": "cluster-config/production",
"path.basename": "production",
"path[0]": "cluster-config",
"path[1]": "production",
"path.basenameNormalized": "production",
"path.filename": "config.json",
"path.filenameNormalized": "config.json",
},
{
"cluster.owner": "john.doe@example.com",
@@ -314,7 +345,10 @@ func TestGitGenerateParamsFromFiles(t *testing.T) {
"path": "cluster-config/production",
"path.basename": "production",
"path[0]": "cluster-config",
"path[1]": "production",
"path.basenameNormalized": "production",
"path.filename": "config.json",
"path.filenameNormalized": "config.json",
},
},
expectedError: nil,
@@ -353,7 +387,10 @@ cluster:
"path": "cluster-config/production",
"path.basename": "production",
"path[0]": "cluster-config",
"path[1]": "production",
"path.basenameNormalized": "production",
"path.filename": "config.yaml",
"path.filenameNormalized": "config.yaml",
},
{
"cluster.owner": "foo.bar@example.com",
@@ -362,7 +399,10 @@ cluster:
"path": "cluster-config/staging",
"path.basename": "staging",
"path[0]": "cluster-config",
"path[1]": "staging",
"path.basenameNormalized": "staging",
"path.filename": "config.yaml",
"path.filenameNormalized": "config.yaml",
},
},
expectedError: nil,
@@ -393,7 +433,10 @@ cluster:
"path": "cluster-config/production",
"path.basename": "production",
"path[0]": "cluster-config",
"path[1]": "production",
"path.basenameNormalized": "production",
"path.filename": "config.yaml",
"path.filenameNormalized": "config.yaml",
},
{
"cluster.owner": "john.doe@example.com",
@@ -402,7 +445,10 @@ cluster:
"path": "cluster-config/production",
"path.basename": "production",
"path[0]": "cluster-config",
"path[1]": "production",
"path.basenameNormalized": "production",
"path.filename": "config.yaml",
"path.filenameNormalized": "config.yaml",
},
},
expectedError: nil,

View File

@@ -1,6 +1,7 @@
package commands
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
@@ -40,16 +41,7 @@ func newAWSCommand() *cobra.Command {
var command = &cobra.Command{
Use: "aws",
Run: func(c *cobra.Command, args []string) {
sess, err := session.NewSession()
errors.CheckError(err)
stsAPI := sts.New(sess)
if roleARN != "" {
creds := stscreds.NewCredentials(sess, roleARN)
stsAPI = sts.New(sess, &aws.Config{Credentials: creds})
}
request, _ := stsAPI.GetCallerIdentityRequest(&sts.GetCallerIdentityInput{})
request.HTTPRequest.Header.Add(clusterIDHeader, clusterName)
presignedURLString, err := request.Presign(requestPresignParam)
presignedURLString, err := getSignedRequestWithRetry(time.Minute, 5*time.Second, clusterName, roleARN, getSignedRequest)
errors.CheckError(err)
token := v1Prefix + base64.RawURLEncoding.EncodeToString([]byte(presignedURLString))
// Set token expiration to 1 minute before the presigned URL expires for some cushion
@@ -62,6 +54,43 @@ func newAWSCommand() *cobra.Command {
return command
}
type getSignedRequestFunc func(clusterName, roleARN string) (string, error)
func getSignedRequestWithRetry(timeout, interval time.Duration, clusterName, roleARN string, fn getSignedRequestFunc) (string, error) {
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
for {
signed, err := fn(clusterName, roleARN)
if err == nil {
return signed, nil
}
select {
case <-ctx.Done():
return "", fmt.Errorf("timeout while trying to get signed aws request: last error: %s", err)
case <-time.After(interval):
}
}
}
func getSignedRequest(clusterName, roleARN string) (string, error) {
sess, err := session.NewSession()
if err != nil {
return "", fmt.Errorf("error creating new AWS session: %s", err)
}
stsAPI := sts.New(sess)
if roleARN != "" {
creds := stscreds.NewCredentials(sess, roleARN)
stsAPI = sts.New(sess, &aws.Config{Credentials: creds})
}
request, _ := stsAPI.GetCallerIdentityRequest(&sts.GetCallerIdentityInput{})
request.HTTPRequest.Header.Add(clusterIDHeader, clusterName)
signed, err := request.Presign(requestPresignParam)
if err != nil {
return "", fmt.Errorf("error presigning AWS request: %s", err)
}
return signed, nil
}
func formatJSON(token string, expiration time.Time) string {
expirationTimestamp := metav1.NewTime(expiration)
execInput := &clientauthv1beta1.ExecCredential{

View File

@@ -0,0 +1,73 @@
package commands
import (
"fmt"
"testing"
"time"
"github.com/stretchr/testify/assert"
)
func TestGetSignedRequestWithRetry(t *testing.T) {
t.Run("will return signed request on first attempt", func(t *testing.T) {
// given
t.Parallel()
mock := &signedRequestMock{
returnFunc: func(m *signedRequestMock) (string, error) {
return "token", nil
},
}
// when
signed, err := getSignedRequestWithRetry(time.Second, time.Millisecond, "cluster-name", "", mock.getSignedRequestMock)
// then
assert.NoError(t, err)
assert.Equal(t, "token", signed)
})
t.Run("will return signed request on third attempt", func(t *testing.T) {
// given
t.Parallel()
mock := &signedRequestMock{
returnFunc: func(m *signedRequestMock) (string, error) {
if m.getSignedRequestCalls < 3 {
return "", fmt.Errorf("some error")
}
return "token", nil
},
}
// when
signed, err := getSignedRequestWithRetry(time.Second, time.Millisecond, "cluster-name", "", mock.getSignedRequestMock)
// then
assert.NoError(t, err)
assert.Equal(t, "token", signed)
})
t.Run("will return error on timeout", func(t *testing.T) {
// given
t.Parallel()
mock := &signedRequestMock{
returnFunc: func(m *signedRequestMock) (string, error) {
return "", fmt.Errorf("some error")
},
}
// when
signed, err := getSignedRequestWithRetry(time.Second, time.Millisecond, "cluster-name", "", mock.getSignedRequestMock)
// then
assert.Error(t, err)
assert.Equal(t, "", signed)
})
}
type signedRequestMock struct {
getSignedRequestCalls int
returnFunc func(m *signedRequestMock) (string, error)
}
func (m *signedRequestMock) getSignedRequestMock(clusterName, roleARN string) (string, error) {
m.getSignedRequestCalls++
return m.returnFunc(m)
}

View File

@@ -14,6 +14,7 @@ import (
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"google.golang.org/grpc/health/grpc_health_v1"
"k8s.io/apimachinery/pkg/api/resource"
cmdutil "github.com/argoproj/argo-cd/v2/cmd/util"
"github.com/argoproj/argo-cd/v2/common"
@@ -70,15 +71,17 @@ func getSubmoduleEnabled() bool {
func NewCommand() *cobra.Command {
var (
parallelismLimit int64
listenPort int
metricsPort int
otlpAddress string
cacheSrc func() (*reposervercache.Cache, error)
tlsConfigCustomizer tls.ConfigCustomizer
tlsConfigCustomizerSrc func() (tls.ConfigCustomizer, error)
redisClient *redis.Client
disableTLS bool
parallelismLimit int64
listenPort int
metricsPort int
otlpAddress string
cacheSrc func() (*reposervercache.Cache, error)
tlsConfigCustomizer tls.ConfigCustomizer
tlsConfigCustomizerSrc func() (tls.ConfigCustomizer, error)
redisClient *redis.Client
disableTLS bool
maxCombinedDirectoryManifestsSize string
cmpTarExcludedGlobs []string
)
var command = cobra.Command{
Use: cliName,
@@ -98,15 +101,20 @@ func NewCommand() *cobra.Command {
cache, err := cacheSrc()
errors.CheckError(err)
maxCombinedDirectoryManifestsQuantity, err := resource.ParseQuantity(maxCombinedDirectoryManifestsSize)
errors.CheckError(err)
askPassServer := askpass.NewServer()
metricsServer := metrics.NewMetricsServer()
cacheutil.CollectMetrics(redisClient, metricsServer)
server, err := reposerver.NewServer(metricsServer, cache, tlsConfigCustomizer, repository.RepoServerInitConstants{
ParallelismLimit: parallelismLimit,
ParallelismLimit: parallelismLimit,
PauseGenerationAfterFailedGenerationAttempts: getPauseGenerationAfterFailedGenerationAttempts(),
PauseGenerationOnFailureForMinutes: getPauseGenerationOnFailureForMinutes(),
PauseGenerationOnFailureForRequests: getPauseGenerationOnFailureForRequests(),
SubmoduleEnabled: getSubmoduleEnabled(),
MaxCombinedDirectoryManifestsSize: maxCombinedDirectoryManifestsQuantity,
CMPTarExcludedGlobs: cmpTarExcludedGlobs,
}, askPassServer)
errors.CheckError(err)
@@ -182,6 +190,8 @@ func NewCommand() *cobra.Command {
command.Flags().IntVar(&metricsPort, "metrics-port", common.DefaultPortRepoServerMetrics, "Start metrics server on given port")
command.Flags().StringVar(&otlpAddress, "otlp-address", env.StringFromEnv("ARGOCD_REPO_SERVER_OTLP_ADDRESS", ""), "OpenTelemetry collector address to send traces to")
command.Flags().BoolVar(&disableTLS, "disable-tls", env.ParseBoolFromEnv("ARGOCD_REPO_SERVER_DISABLE_TLS", false), "Disable TLS on the gRPC endpoint")
command.Flags().StringVar(&maxCombinedDirectoryManifestsSize, "max-combined-directory-manifests-size", env.StringFromEnv("ARGOCD_REPO_SERVER_MAX_COMBINED_DIRECTORY_MANIFESTS_SIZE", "10M"), "Max combined size of manifest files in a directory-type Application")
command.Flags().StringArrayVar(&cmpTarExcludedGlobs, "plugin-tar-exclude", env.StringsFromEnv("ARGOCD_REPO_SERVER_PLUGIN_TAR_EXCLUSIONS", []string{}, ";"), "Globs to filter when sending tarballs to plugins.")
tlsConfigCustomizerSrc = tls.AddTLSFlagsToCmd(&command)
cacheSrc = reposervercache.AddCacheFlagsToCmd(&command, func(client *redis.Client) {

View File

@@ -152,7 +152,10 @@ func NewCommand() *cobra.Command {
stats.RegisterStackDumper()
stats.StartStatsTicker(10 * time.Minute)
stats.RegisterHeapDumper("memprofile")
argocd := server.NewServer(context.Background(), argoCDOpts)
argocd.Init(context.Background())
lns, err := argocd.Listen()
errors.CheckError(err)
for {
var closer func()
ctx := context.Background()
@@ -163,8 +166,7 @@ func NewCommand() *cobra.Command {
log.Fatalf("failed to initialize tracing: %v", err)
}
}
argocd := server.NewServer(ctx, argoCDOpts)
argocd.Run(ctx, listenPort, metricsPort)
argocd.Run(ctx, lns)
cancel()
if closer != nil {
closer()

View File

@@ -25,6 +25,7 @@ import (
"github.com/spf13/cobra"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
"k8s.io/apimachinery/pkg/api/resource"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/apimachinery/pkg/runtime/schema"
"k8s.io/apimachinery/pkg/types"
@@ -778,7 +779,7 @@ func getLocalObjectsString(app *argoappv1.Application, local, localRepoRoot, app
ApiVersions: apiVersions,
Plugins: configManagementPlugins,
TrackingMethod: trackingMethod,
}, true, &git.NoopCredsStore{})
}, true, &git.NoopCredsStore{}, resource.MustParse("0"))
errors.CheckError(err)
return res.Manifests

View File

@@ -36,13 +36,13 @@ users:
- auth-token: vErrYS3c3tReFRe$hToken
name: localhost:8080`
const testConfigFilePath = "./testdata/config"
const testConfigFilePath = "./testdata/local.config"
func TestContextDelete(t *testing.T) {
// Write the test config file
err := ioutil.WriteFile(testConfigFilePath, []byte(testConfig), os.ModePerm)
assert.NoError(t, err)
defer os.Remove(testConfigFilePath)
err = os.Chmod(testConfigFilePath, 0600)
require.NoError(t, err, "Could not change the file permission to 0600 %v", err)
@@ -75,9 +75,4 @@ func TestContextDelete(t *testing.T) {
assert.NotContains(t, localConfig.Servers, localconfig.Server{PlainText: true, Server: "localhost:8080"})
assert.NotContains(t, localConfig.Users, localconfig.User{AuthToken: "vErrYS3c3tReFRe$hToken", Name: "localhost:8080"})
assert.Contains(t, localConfig.Contexts, localconfig.ContextRef{Name: "argocd2.example.com:443", Server: "argocd2.example.com:443", User: "argocd2.example.com:443"})
// Write the file again so that no conflicts are made in git
err = ioutil.WriteFile(testConfigFilePath, []byte(testConfig), os.ModePerm)
assert.NoError(t, err)
}

View File

@@ -215,8 +215,13 @@ func StartLocalServer(clientOpts *apiclient.ClientOptions, ctxStr string, port *
ListenHost: *address,
RepoClientset: &forwardRepoClientset{namespace: namespace, context: ctxStr},
})
srv.Init(ctx)
go srv.Run(ctx, *port, 0)
lns, err := srv.Listen()
if err != nil {
return err
}
go srv.Run(ctx, lns)
clientOpts.ServerAddr = fmt.Sprintf("%s:%d", *address, *port)
clientOpts.PlainText = true
if !cache2.WaitForCacheSync(ctx.Done(), srv.Initialized) {

View File

@@ -202,7 +202,10 @@ func oauth2Login(ctx context.Context, port int, oidcSettings *settingspkg.OIDCCo
// completionChan is to signal flow completed. Non-empty string indicates error
completionChan := make(chan string)
// stateNonce is an OAuth2 state nonce
stateNonce := rand.RandString(10)
// According to the spec (https://www.rfc-editor.org/rfc/rfc6749#section-10.10), this must be guessable with
// probability <= 2^(-128). The following call generates one of 52^24 random strings, ~= 2^136 possibilities.
stateNonce, err := rand.String(24)
errors.CheckError(err)
var tokenString string
var refreshToken string
@@ -212,7 +215,8 @@ func oauth2Login(ctx context.Context, port int, oidcSettings *settingspkg.OIDCCo
}
// PKCE implementation of https://tools.ietf.org/html/rfc7636
codeVerifier := rand.RandStringCharset(43, "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~")
codeVerifier, err := rand.StringFromCharset(43, "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~")
errors.CheckError(err)
codeChallengeHash := sha256.Sum256([]byte(codeVerifier))
codeChallenge := base64.RawURLEncoding.EncodeToString(codeChallengeHash[:])
@@ -296,7 +300,8 @@ func oauth2Login(ctx context.Context, port int, oidcSettings *settingspkg.OIDCCo
opts = append(opts, oauth2.SetAuthURLParam("code_challenge_method", "S256"))
url = oauth2conf.AuthCodeURL(stateNonce, opts...)
case oidcutil.GrantTypeImplicit:
url = oidcutil.ImplicitFlowURL(oauth2conf, stateNonce, opts...)
url, err = oidcutil.ImplicitFlowURL(oauth2conf, stateNonce, opts...)
errors.CheckError(err)
default:
log.Fatalf("Unsupported grant type: %v", grantType)
}

View File

@@ -6,6 +6,7 @@ import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
argocdclient "github.com/argoproj/argo-cd/v2/pkg/apiclient"
"github.com/argoproj/argo-cd/v2/util/localconfig"
@@ -16,6 +17,10 @@ func TestLogout(t *testing.T) {
// Write the test config file
err := ioutil.WriteFile(testConfigFilePath, []byte(testConfig), os.ModePerm)
assert.NoError(t, err)
defer os.Remove(testConfigFilePath)
err = os.Chmod(testConfigFilePath, 0600)
require.NoError(t, err)
localConfig, err := localconfig.ReadLocalConfig(testConfigFilePath)
assert.NoError(t, err)
@@ -32,9 +37,4 @@ func TestLogout(t *testing.T) {
assert.Contains(t, localConfig.Contexts, localconfig.ContextRef{Name: "argocd1.example.com:443", Server: "argocd1.example.com:443", User: "argocd1.example.com:443"})
assert.Contains(t, localConfig.Contexts, localconfig.ContextRef{Name: "argocd2.example.com:443", Server: "argocd2.example.com:443", User: "argocd2.example.com:443"})
assert.Contains(t, localConfig.Contexts, localconfig.ContextRef{Name: "localhost:8080", Server: "localhost:8080", User: "localhost:8080"})
// Write the file again so that no conflicts are made in git
err = ioutil.WriteFile(testConfigFilePath, []byte(testConfig), os.ModePerm)
assert.NoError(t, err)
}

View File

@@ -195,7 +195,8 @@ func TestMatchRepository(t *testing.T) {
d := Discover{
Find: Find{
Command: Command{
Command: []string{"sh", "-c", "echo -n $ENV_NO_VAR"},
// Use printf instead of echo since OSX prints the "-n" when there's no additional arg.
Command: []string{"sh", "-c", `printf "%s" "$ENV_NO_VAR"`},
},
},
}

View File

@@ -149,7 +149,13 @@ func (m *appStateManager) SyncAppState(app *v1alpha1.Application, state *v1alpha
}
atomic.AddUint64(&syncIdPrefix, 1)
syncId := fmt.Sprintf("%05d-%s", syncIdPrefix, rand.RandString(5))
randSuffix, err := rand.String(5)
if err != nil {
state.Phase = common.OperationError
state.Message = fmt.Sprintf("Failed generate random sync ID: %v", err)
return
}
syncId := fmt.Sprintf("%05d-%s", syncIdPrefix, randSuffix)
logEntry := log.WithFields(log.Fields{"application": app.Name, "syncId": syncId})
initialResourcesRes := make([]common.ResourceSyncResult, 0)

View File

@@ -4,10 +4,10 @@ You can download the latest Argo CD version from [the latest release page of thi
## Linux and WSL
### ArchLinux User Repository ([AUR](https://aur.archlinux.org/packages/))
### ArchLinux
```bash
yay -Sy argocd-bin
pacman -S argocd
```
### Homebrew

View File

@@ -65,13 +65,15 @@ The generator parameters are:
- `{{path.basename}}`: For any directory path within the Git repository that matches the `path` wildcard, the right-most path name is extracted (e.g. `/directory/directory2` would produce `directory2`).
- `{{path.basenameNormalized}}`: This field is the same as `path.basename` with unsupported characters replaced with `-` (e.g. a `path` of `/directory/directory_2`, and `path.basename` of `directory_2` would produce `directory-2` here).
Whenever a new Helm chart/Kustomize YAML/Application/plain subfolder is added to the Git repository, the ApplicationSet controller will detect this change and automatically deploy the resulting manifests within new `Application` resources.
**Note**: The right-most path name always becomes `{{path.basename}}`. For example, for `- path: /one/two/three/four`, `{{path.basename}}` is `four`.
Whenever a new Helm chart/Kustomize YAML/Application/plain subdirectory is added to the Git repository, the ApplicationSet controller will detect this change and automatically deploy the resulting manifests within new `Application` resources.
As with other generators, clusters *must* already be defined within Argo CD, in order to generate Applications for them.
### Exclude directories
The Git directory generator will automatically exclude folders that begin with `.` (such as `.git`).
The Git directory generator will automatically exclude directories that begin with `.` (such as `.git`).
The Git directory generator also supports an `exclude` option in order to exclude directories in the repository from being scanned by the ApplicationSet controller:
@@ -150,6 +152,41 @@ Or, a shorter way (using [path.Match](https://golang.org/pkg/path/#Match) syntax
exclude: true
```
### Root Of Git Repo
The Git directory generator can be configured to deploy from the root of the git repository by providing `'*'` as the `path`.
To exclude directories, you only need to put the name/[path.Match](https://golang.org/pkg/path/#Match) of the directory you do not want to deploy.
```yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: cluster-addons
namespace: argocd
spec:
generators:
- git:
repoURL: https://github.com/example/example-repo.git
revision: HEAD
directories:
- path: '*'
- path: donotdeploy
exclude: true
template:
metadata:
name: '{{path.basename}}'
spec:
project: "my-project"
source:
repoURL: https://github.com/example/example-repo.git
targetRevision: HEAD
path: '{{path}}'
destination:
server: https://kubernetes.default.svc
namespace: '{{path.basename}}'
```
## Git Generator: Files
The Git file generator is the second subtype of the Git generator. The Git file generator generates parameters using the contents of JSON/YAML files found within a specified repository.
@@ -170,7 +207,7 @@ Suppose you have a Git repository with the following directory structure:
└── git-generator-files.yaml
```
The folders are:
The directories are:
- `guestbook` contains the Kubernetes resources for a simple guestbook application
- `cluster-config` contains JSON/YAML files describing the individual engineering clusters: one for `dev` and one for `prod`.
@@ -234,10 +271,16 @@ As with other generators, clusters *must* already be defined within Argo CD, in
In addition to the flattened key/value pairs from the configuration file, the following generator parameters are provided:
- `{{path}}`: The path to the folder containing matching configuration file within the Git repository. Example: `/clusters/clusterA`, if the config file was `/clusters/clusterA/config.json`
- `{{path}}`: The path to the directory containing matching configuration file within the Git repository. Example: `/clusters/clusterA`, if the config file was `/clusters/clusterA/config.json`
- `{{path[n]}}`: The path to the matching configuration file within the Git repository, split into array elements (`n` - array index). Example: `path[0]: clusters`, `path[1]: clusterA`
- `{{path.basename}}`: Basename of the path to the folder containing the configuration file (e.g. `clusterA`, with the above example.)
- `{{path.basename}}`: Basename of the path to the directory containing the configuration file (e.g. `clusterA`, with the above example.)
- `{{path.basenameNormalized}}`: This field is the same as `path.basename` with unsupported characters replaced with `-` (e.g. a `path` of `/directory/directory_2`, and `path.basename` of `directory_2` would produce `directory-2` here).
- `{{path.filename}}`: The matched filename. e.g., `config.json` in the above example.
- `{{path.filenameNormalized}}`: The matched filename with unsupported characters replaced with `-`.
**Note**: The right-most *directory* name always becomes `{{path.basename}}`. For example, from `- path: /one/two/three/four/config.json`, `{{path.basename}}` will be `four`.
The filename can always be accessed using `{{path.filename}}`.
## Webhook Configuration

View File

@@ -106,4 +106,10 @@ data:
reposerver.repo.cache.expiration: "24h0m0s"
# Cache expiration default (default 24h0m0s)
reposerver.default.cache.expiration: "24h0m0s"
# Max combined manifest file size for a single directory-type Application. In-memory manifest representation may be as
# much as 300x the manifest file size. Limit this to stay within the memory limits of the repo-server while allowing
# for 300x memory expansion and N Applications running at the same time.
# (example 10M max * 300 expansion * 10 Apps = 30G max theoretical memory usage).
reposerver.max.combined.directory.manifests.size: '10M'
# Paths to be excluded from the tarball streamed to plugins. Separate with ;
reposerver.plugin.tar.exclusions: ""

View File

@@ -223,3 +223,44 @@ to configure IP addresses logging in the proxy server that sits in front of the
Argo CD's ApplicationSets feature has its own [security considerations](./applicationset/Security.md). Be aware of those
issues before using ApplicationSets.
## Limiting Directory App Memory Usage
> >2.2.10, 2.1.16, >2.3.5
Directory-type Applications (those whose source is raw JSON or YAML files) can consume significant
[repo-server](architecture.md#repository-server) memory, depending on the size and structure of the YAML files.
To avoid over-using memory in the repo-server (potentially causing a crash and denial of service), set the
`reposerver.max.combined.directory.manifests.size` config option in [argocd-cmd-params-cm](argocd-cmd-params-cm.yaml).
This option limits the combined size of all JSON or YAML files in an individual app. Note that the in-memory
representation of a manifest may be as much as 300x the size of the manifest on disk. Also note that the limit is per
Application. If manifests are generated for multiple applications at once, memory usage will be higher.
**Example:**
Suppose your repo-server has a 10G memory limit, and you have ten Applications which use raw JSON or YAML files. To
calculate the max safe combined file size per Application, divide 10G by 300 * 10 Apps (300 being the worst-case memory
growth factor for the manifests).
```
10G / 300 * 10 = 3M
```
So a reasonably safe configuration for this setup would be a 3M limit per app.
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-cmd-params-cm
data:
reposerver.max.combined.directory.manifests.size: '3M'
```
The 300x ratio assumes a maliciously-crafted manifest file. If you only want to protect against accidental excessive
memory use, it is probably safe to use a smaller ratio.
Keep in mind that if a malicious user can create additional Applications, they can increase the total memory usage.
Grant [App creation privileges](rbac.md) carefully.

View File

@@ -13,28 +13,30 @@ argocd-repo-server [flags]
### Options
```
--default-cache-expiration duration Cache expiration default (default 24h0m0s)
--disable-tls Disable TLS on the gRPC endpoint
-h, --help help for argocd-repo-server
--logformat string Set the logging format. One of: text|json (default "text")
--loglevel string Set the logging level. One of: debug|info|warn|error (default "info")
--metrics-port int Start metrics server on given port (default 8084)
--otlp-address string OpenTelemetry collector address to send traces to
--parallelismlimit int Limit on number of concurrent manifests generate requests. Any value less the 1 means no limit.
--port int Listen on given port for incoming connections (default 8081)
--redis string Redis server hostname and port (e.g. argocd-redis:6379).
--redis-ca-certificate string Path to Redis server CA certificate (e.g. /etc/certs/redis/ca.crt). If not specified, system trusted CAs will be used for server certificate validation.
--redis-client-certificate string Path to Redis client certificate (e.g. /etc/certs/redis/client.crt).
--redis-client-key string Path to Redis client key (e.g. /etc/certs/redis/client.crt).
--redis-insecure-skip-tls-verify Skip Redis server certificate validation.
--redis-use-tls Use TLS when connecting to Redis.
--redisdb int Redis database.
--repo-cache-expiration duration Cache expiration for repo state, incl. app lists, app details, manifest generation, revision meta-data (default 24h0m0s)
--revision-cache-expiration duration Cache expiration for cached revision (default 3m0s)
--sentinel stringArray Redis sentinel hostname and port (e.g. argocd-redis-ha-announce-0:6379).
--sentinelmaster string Redis sentinel master group name. (default "master")
--tlsciphers string The list of acceptable ciphers to be used when establishing TLS connections. Use 'list' to list available ciphers. (default "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384:TLS_RSA_WITH_AES_256_GCM_SHA384")
--tlsmaxversion string The maximum SSL/TLS version that is acceptable (one of: 1.0|1.1|1.2|1.3) (default "1.3")
--tlsminversion string The minimum SSL/TLS version that is acceptable (one of: 1.0|1.1|1.2|1.3) (default "1.2")
--default-cache-expiration duration Cache expiration default (default 24h0m0s)
--disable-tls Disable TLS on the gRPC endpoint
-h, --help help for argocd-repo-server
--logformat string Set the logging format. One of: text|json (default "text")
--loglevel string Set the logging level. One of: debug|info|warn|error (default "info")
--max-combined-directory-manifests-size string Max combined size of manifest files in a directory-type Application (default "10M")
--metrics-port int Start metrics server on given port (default 8084)
--otlp-address string OpenTelemetry collector address to send traces to
--parallelismlimit int Limit on number of concurrent manifests generate requests. Any value less the 1 means no limit.
--plugin-tar-exclude stringArray Globs to filter when sending tarballs to plugins.
--port int Listen on given port for incoming connections (default 8081)
--redis string Redis server hostname and port (e.g. argocd-redis:6379).
--redis-ca-certificate string Path to Redis server CA certificate (e.g. /etc/certs/redis/ca.crt). If not specified, system trusted CAs will be used for server certificate validation.
--redis-client-certificate string Path to Redis client certificate (e.g. /etc/certs/redis/client.crt).
--redis-client-key string Path to Redis client key (e.g. /etc/certs/redis/client.crt).
--redis-insecure-skip-tls-verify Skip Redis server certificate validation.
--redis-use-tls Use TLS when connecting to Redis.
--redisdb int Redis database.
--repo-cache-expiration duration Cache expiration for repo state, incl. app lists, app details, manifest generation, revision meta-data (default 24h0m0s)
--revision-cache-expiration duration Cache expiration for cached revision (default 3m0s)
--sentinel stringArray Redis sentinel hostname and port (e.g. argocd-redis-ha-announce-0:6379).
--sentinelmaster string Redis sentinel master group name. (default "master")
--tlsciphers string The list of acceptable ciphers to be used when establishing TLS connections. Use 'list' to list available ciphers. (default "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384:TLS_RSA_WITH_AES_256_GCM_SHA384")
--tlsmaxversion string The maximum SSL/TLS version that is acceptable (one of: 1.0|1.1|1.2|1.3) (default "1.3")
--tlsminversion string The minimum SSL/TLS version that is acceptable (one of: 1.0|1.1|1.2|1.3) (default "1.2")
```

View File

@@ -197,6 +197,7 @@ NOTES:
* There is no need to set `redirectURI` in the `connectors.config` as shown in the dex documentation.
Argo CD will automatically use the correct `redirectURI` for any OAuth2 connectors, to match the
correct external callback URL (e.g. `https://argocd.example.com/api/dex/callback`)
* When using a custom secret (e.g., `some_K8S_secret` above,) it *must* have the label `app.kubernetes.io/part-of: argocd`.
## OIDC Configuration with DEX

View File

@@ -234,3 +234,17 @@ If you don't need to set any environment variables, you can set an empty plugin
Each CMP command will also independently timeout on the `ARGOCD_EXEC_TIMEOUT` set for the CMP sidecar. The default
is 90s. So if you increase the repo server timeout greater than 90s, be sure to set `ARGOCD_EXEC_TIMEOUT` on the
sidecar.
## Tarball stream filtering
In order to increase the speed of manifest generation, certain files and folders can be excluded from being sent to your
plugin. We recommend excluding your `.git` folder if it isn't necessary. Use Go's
[filepatch.Match](https://pkg.go.dev/path/filepath#Match) syntax.
You can set it one of three ways:
1. The `--plugin-tar-exclude` argument on the repo server.
2. The `reposerver.plugin.tar.exclusions` key if you are using `argocd-cmd-params-cm`
3. Directly setting 'ARGOCD_REPO_SERVER_PLUGIN_TAR_EXCLUSIONS' environment variable on the repo server.
For option 1, the flag can be repeated multiple times. For option 2 and 3, you can specify multiple globs by separating
them with semicolons.

View File

@@ -0,0 +1,9 @@
# Environment Variables
The following environment variables can be used with `argocd` CLI:
| Environment Variable | Description |
| --- | --- |
| `ARGOCD_SERVER` | the address of the ArgoCD server without `https://` prefix <br> (instead of specifying `--server` for every command) <br> eg. `ARGOCD_SERVER=argocd.mycompany.com` if served through an ingress with DNS |
| `ARGOCD_AUTH_TOKEN` | the ArgoCD `apiKey` for your ArgoCD user to be able to authenticate |
| `ARGOCD_OPTS` | command-line options to pass to `argocd` CLI <br> eg. `ARGOCD_OPTS="--grpc-web"` |

View File

@@ -2,7 +2,7 @@
ArgoCD allows users to customize some aspects of how it syncs the desired state in the target cluster. Some Sync Options can defined as annotations in a specific resource. Most of the Sync Options are configured in the Application resource `spec.syncPolicy.syncOptions` attribute.
Bellow you can find details about each available Sync Option:
Below you can find details about each available Sync Option:
## No Prune Resources

14
go.mod
View File

@@ -17,7 +17,7 @@ require (
github.com/bradleyfalzon/ghinstallation/v2 v2.0.4
github.com/casbin/casbin/v2 v2.39.1
github.com/chai2010/gettext-go v0.0.0-20170215093142-bf70f2a70fb1 // indirect
github.com/coreos/go-oidc v2.1.0+incompatible
github.com/coreos/go-oidc v2.2.1+incompatible
github.com/dustin/go-humanize v1.0.0
github.com/evanphx/json-patch v4.12.0+incompatible
github.com/fsnotify/fsnotify v1.5.1
@@ -60,7 +60,7 @@ require (
github.com/olekukonko/tablewriter v0.0.5
github.com/patrickmn/go-cache v2.1.0+incompatible
github.com/pkg/errors v0.9.1
github.com/pquerna/cachecontrol v0.0.0-20180306154005-525d0eb5f91d // indirect
github.com/pquerna/cachecontrol v0.1.0 // indirect
github.com/prometheus/client_golang v1.11.0
github.com/r3labs/diff v1.1.0
github.com/robfig/cron v1.2.0
@@ -76,10 +76,10 @@ require (
github.com/xanzy/go-gitlab v0.60.0
github.com/yuin/gopher-lua v0.0.0-20200816102855-ee81675732da
golang.org/x/crypto v0.0.0-20220525230936-793ad666bf5e
golang.org/x/net v0.0.0-20211209124913-491a49abca63
golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8
golang.org/x/net v0.0.0-20220621193019-9d032be2e588
golang.org/x/oauth2 v0.0.0-20220608161450-d0670ef3b1eb
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211
google.golang.org/genproto v0.0.0-20211208223120-3a66f561d7aa
google.golang.org/grpc v1.45.0
google.golang.org/protobuf v1.28.0
@@ -202,7 +202,7 @@ require (
go.starlark.net v0.0.0-20200306205701-8dd3e2ee1dd5 // indirect
golang.org/x/exp v0.0.0-20210901193431-a062eea981d2 // indirect
golang.org/x/mod v0.5.1-0.20210830214625-1b1db11ec8f4 // indirect
golang.org/x/sys v0.0.0-20211205182925-97ca703d548d // indirect
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a // indirect
golang.org/x/text v0.3.7 // indirect
golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac // indirect
golang.org/x/tools v0.1.6-0.20210820212750-d4cc65f0b2ff // indirect
@@ -214,7 +214,7 @@ require (
gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc // indirect
gopkg.in/gomail.v2 v2.0.0-20160411212932-81ebce5c23df // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/square/go-jose.v2 v2.2.2 // indirect
gopkg.in/square/go-jose.v2 v2.6.0 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect
k8s.io/apiserver v0.23.1

24
go.sum
View File

@@ -254,8 +254,9 @@ github.com/coredns/corefile-migration v1.0.14/go.mod h1:XnhgULOEouimnzgn0t4WPuFD
github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
github.com/coreos/etcd v3.3.13+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
github.com/coreos/go-oidc v2.1.0+incompatible h1:sdJrfw8akMnCuUlaZU3tE/uYXFgfqom8DBE9so9EBsM=
github.com/coreos/go-oidc v2.1.0+incompatible/go.mod h1:CgnwVTmzoESiwO9qyAFEMiHoZ1nMCKZlZ9V6mm3/LKc=
github.com/coreos/go-oidc v2.2.1+incompatible h1:mh48q/BqXqgjVHpy2ZY7WnWAbenxRjsz9N1i1YxjHAk=
github.com/coreos/go-oidc v2.2.1+incompatible/go.mod h1:CgnwVTmzoESiwO9qyAFEMiHoZ1nMCKZlZ9V6mm3/LKc=
github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
github.com/coreos/go-systemd v0.0.0-20180511133405-39ca1b05acc7/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
@@ -944,8 +945,8 @@ github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZN
github.com/posener/complete v1.1.1/go.mod h1:em0nMJCgc9GFtwrmVmEMR/ZL6WyhyjMBndrE9hABlRI=
github.com/posener/complete v1.2.3/go.mod h1:WZIdtGGp+qx0sLrYKtIRAruyNpv6hFCicSgv7Sy7s/s=
github.com/pquerna/cachecontrol v0.0.0-20171018203845-0dec1b30a021/go.mod h1:prYjPmNq4d1NPVmpShWobRqXY3q7Vp+80DqgxxUrUIA=
github.com/pquerna/cachecontrol v0.0.0-20180306154005-525d0eb5f91d h1:7gXyC293Lsm2YWgQ+0uaAFFFDO82ruiQSwc3ua+Vtlc=
github.com/pquerna/cachecontrol v0.0.0-20180306154005-525d0eb5f91d/go.mod h1:prYjPmNq4d1NPVmpShWobRqXY3q7Vp+80DqgxxUrUIA=
github.com/pquerna/cachecontrol v0.1.0 h1:yJMy84ti9h/+OEWa752kBTKv4XC30OtVVHYv/8cTqKc=
github.com/pquerna/cachecontrol v0.1.0/go.mod h1:NrUG3Z7Rdu85UNR3vm7SOsl1nFIeSiQnrHV5K9mBcUI=
github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
github.com/prometheus/client_golang v0.9.3-0.20190127221311-3c4408c8b829/go.mod h1:p2iRAGwDERtqlqzRXnrOVns+ignqQo//hLXqYxZYVNs=
github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso=
@@ -1343,8 +1344,10 @@ golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qx
golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20210813160813-60bc85c4be6d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20210825183410-e898025ed96a/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211209124913-491a49abca63 h1:iocB37TsdFuN6IBRZ+ry36wrkoV51/tl5vOWqkcPGvY=
golang.org/x/net v0.0.0-20211209124913-491a49abca63/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
golang.org/x/net v0.0.0-20220621193019-9d032be2e588 h1:9ubFuySsnAJYGyJrZ3koiEv8FyqofCBdz3G9Mbf2YFc=
golang.org/x/net v0.0.0-20220621193019-9d032be2e588/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/oauth2 v0.0.0-20180227000427-d7d64896b5ff/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -1364,8 +1367,9 @@ golang.org/x/oauth2 v0.0.0-20210628180205-a41e5a781914/go.mod h1:KelEdhl1UZF7XfJ
golang.org/x/oauth2 v0.0.0-20210805134026-6f1e6394065a/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20210819190943-2bc19b11175f/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20211005180243-6b3c2da341f1/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8 h1:RerP+noqYHUQ8CMRcPlC2nvTa4dcBIjegkuWdcUDuqg=
golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20220608161450-d0670ef3b1eb h1:8tDJ3aechhddbdPAxpycgXHJRMLpk/Ab+aa4OgdN5/g=
golang.org/x/oauth2 v0.0.0-20220608161450-d0670ef3b1eb/go.mod h1:jaDAt6Dkxork7LmZnYtzbRWj0W47D86a3TGe0YHBvmE=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -1485,11 +1489,14 @@ golang.org/x/sys v0.0.0-20210908233432-aa78b53d3365/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211007075335-d3039528d8ac/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211124211545-fe61309f8881/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211205182925-97ca703d548d h1:FjkYO/PPp4Wi0EAUOVLxePm7qVW4r4ctbWpURyuOD0E=
golang.org/x/sys v0.0.0-20211205182925-97ca703d548d/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a h1:dGzPydgVsqGcTRVwiLJ1jVbufYwmzD3LfVPLKsKg+0k=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b h1:9zKuko04nR4gjZ4+DNjHqRlAJqbJETHwiNKDqTfOjfE=
golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211 h1:JGgROgKl9N8DuW20oFS5gxc+lE67/N3FcwmBPMe7ArY=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -1789,8 +1796,9 @@ gopkg.in/ini.v1 v1.62.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
gopkg.in/ini.v1 v1.66.2/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
gopkg.in/mgo.v2 v2.0.0-20180705113604-9856a29383ce/go.mod h1:yeKp02qBN3iKW1OzL3MGk2IdtZzaj7SFntXj72NppTA=
gopkg.in/natefinch/lumberjack.v2 v2.0.0/go.mod h1:l0ndWWf7gzL7RNwBG7wST/UCcT4T24xpD6X8LsfU/+k=
gopkg.in/square/go-jose.v2 v2.2.2 h1:orlkJ3myw8CN1nVQHBFfloD+L3egixIa4FvUP6RosSA=
gopkg.in/square/go-jose.v2 v2.2.2/go.mod h1:M9dMgbHiYLoDGQrXy7OpJDJWiKiU//h+vD76mk0e1AI=
gopkg.in/square/go-jose.v2 v2.6.0 h1:NGk74WTnPKBNUhNzQX7PYcTLUjoq7mzKk2OKbvwk2iI=
gopkg.in/square/go-jose.v2 v2.6.0/go.mod h1:M9dMgbHiYLoDGQrXy7OpJDJWiKiU//h+vD76mk0e1AI=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
gopkg.in/warnings.v0 v0.1.1/go.mod h1:jksf8JmL6Qr/oQM2OXTHunEvvTAsrWBLb6OOjuVWRNI=

View File

@@ -5,7 +5,7 @@ kind: Kustomization
images:
- name: quay.io/argoproj/argocd
newName: quay.io/argoproj/argocd
newTag: v2.4.0
newTag: v2.4.4
resources:
- ./application-controller
- ./dex

View File

@@ -101,6 +101,18 @@ spec:
name: argocd-cmd-params-cm
key: otlp.address
optional: true
- name: ARGOCD_REPO_SERVER_MAX_COMBINED_DIRECTORY_MANIFESTS_SIZE
valueFrom:
configMapKeyRef:
name: argocd-cmd-params-cm
key: reposerver.max.combined.directory.manifests.size
optional: true
- name: ARGOCD_REPO_SERVER_PLUGIN_TAR_EXCLUSIONS
valueFrom:
configMapKeyRef:
name: argocd-cmd-params-cm
key: reposerver.plugin.tar.exclusions
optional: true
- name: HELM_CACHE_HOME
value: /helm-working-dir
- name: HELM_CONFIG_HOME

View File

@@ -9385,7 +9385,7 @@ spec:
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
name: argocd-applicationset-controller
ports:
@@ -9597,13 +9597,25 @@ spec:
key: otlp.address
name: argocd-cmd-params-cm
optional: true
- name: ARGOCD_REPO_SERVER_MAX_COMBINED_DIRECTORY_MANIFESTS_SIZE
valueFrom:
configMapKeyRef:
key: reposerver.max.combined.directory.manifests.size
name: argocd-cmd-params-cm
optional: true
- name: ARGOCD_REPO_SERVER_PLUGIN_TAR_EXCLUSIONS
valueFrom:
configMapKeyRef:
key: reposerver.plugin.tar.exclusions
name: argocd-cmd-params-cm
optional: true
- name: HELM_CACHE_HOME
value: /helm-working-dir
- name: HELM_CONFIG_HOME
value: /helm-working-dir
- name: HELM_DATA_HOME
value: /helm-working-dir
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
livenessProbe:
failureThreshold: 3
@@ -9652,7 +9664,7 @@ spec:
- -n
- /usr/local/bin/argocd
- /var/run/argocd/argocd-cmp-server
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
name: copyutil
securityContext:
allowPrivilegeEscalation: false
@@ -9839,7 +9851,7 @@ spec:
key: otlp.address
name: argocd-cmd-params-cm
optional: true
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
livenessProbe:
httpGet:

View File

@@ -12,4 +12,4 @@ resources:
images:
- name: quay.io/argoproj/argocd
newName: quay.io/argoproj/argocd
newTag: v2.4.0
newTag: v2.4.4

View File

@@ -11,7 +11,7 @@ patchesStrategicMerge:
images:
- name: quay.io/argoproj/argocd
newName: quay.io/argoproj/argocd
newTag: v2.4.0
newTag: v2.4.4
resources:
- ../../base/application-controller
- ../../base/applicationset-controller

View File

@@ -10320,7 +10320,7 @@ spec:
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
name: argocd-applicationset-controller
ports:
@@ -10417,7 +10417,7 @@ spec:
- -n
- /usr/local/bin/argocd
- /shared/argocd-dex
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
name: copyutil
securityContext:
@@ -10457,7 +10457,7 @@ spec:
containers:
- command:
- argocd-notifications
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
livenessProbe:
tcpSocket:
@@ -10696,13 +10696,25 @@ spec:
key: otlp.address
name: argocd-cmd-params-cm
optional: true
- name: ARGOCD_REPO_SERVER_MAX_COMBINED_DIRECTORY_MANIFESTS_SIZE
valueFrom:
configMapKeyRef:
key: reposerver.max.combined.directory.manifests.size
name: argocd-cmd-params-cm
optional: true
- name: ARGOCD_REPO_SERVER_PLUGIN_TAR_EXCLUSIONS
valueFrom:
configMapKeyRef:
key: reposerver.plugin.tar.exclusions
name: argocd-cmd-params-cm
optional: true
- name: HELM_CACHE_HOME
value: /helm-working-dir
- name: HELM_CONFIG_HOME
value: /helm-working-dir
- name: HELM_DATA_HOME
value: /helm-working-dir
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
livenessProbe:
failureThreshold: 3
@@ -10751,7 +10763,7 @@ spec:
- -n
- /usr/local/bin/argocd
- /var/run/argocd/argocd-cmp-server
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
name: copyutil
securityContext:
allowPrivilegeEscalation: false
@@ -10998,7 +11010,7 @@ spec:
key: otlp.address
name: argocd-cmd-params-cm
optional: true
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
livenessProbe:
httpGet:
@@ -11206,7 +11218,7 @@ spec:
key: otlp.address
name: argocd-cmd-params-cm
optional: true
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
livenessProbe:
httpGet:

View File

@@ -1244,7 +1244,7 @@ spec:
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
name: argocd-applicationset-controller
ports:
@@ -1341,7 +1341,7 @@ spec:
- -n
- /usr/local/bin/argocd
- /shared/argocd-dex
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
name: copyutil
securityContext:
@@ -1381,7 +1381,7 @@ spec:
containers:
- command:
- argocd-notifications
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
livenessProbe:
tcpSocket:
@@ -1620,13 +1620,25 @@ spec:
key: otlp.address
name: argocd-cmd-params-cm
optional: true
- name: ARGOCD_REPO_SERVER_MAX_COMBINED_DIRECTORY_MANIFESTS_SIZE
valueFrom:
configMapKeyRef:
key: reposerver.max.combined.directory.manifests.size
name: argocd-cmd-params-cm
optional: true
- name: ARGOCD_REPO_SERVER_PLUGIN_TAR_EXCLUSIONS
valueFrom:
configMapKeyRef:
key: reposerver.plugin.tar.exclusions
name: argocd-cmd-params-cm
optional: true
- name: HELM_CACHE_HOME
value: /helm-working-dir
- name: HELM_CONFIG_HOME
value: /helm-working-dir
- name: HELM_DATA_HOME
value: /helm-working-dir
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
livenessProbe:
failureThreshold: 3
@@ -1675,7 +1687,7 @@ spec:
- -n
- /usr/local/bin/argocd
- /var/run/argocd/argocd-cmp-server
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
name: copyutil
securityContext:
allowPrivilegeEscalation: false
@@ -1922,7 +1934,7 @@ spec:
key: otlp.address
name: argocd-cmd-params-cm
optional: true
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
livenessProbe:
httpGet:
@@ -2130,7 +2142,7 @@ spec:
key: otlp.address
name: argocd-cmd-params-cm
optional: true
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
livenessProbe:
httpGet:

View File

@@ -9692,7 +9692,7 @@ spec:
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
name: argocd-applicationset-controller
ports:
@@ -9789,7 +9789,7 @@ spec:
- -n
- /usr/local/bin/argocd
- /shared/argocd-dex
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
name: copyutil
securityContext:
@@ -9829,7 +9829,7 @@ spec:
containers:
- command:
- argocd-notifications
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
livenessProbe:
tcpSocket:
@@ -10036,13 +10036,25 @@ spec:
key: otlp.address
name: argocd-cmd-params-cm
optional: true
- name: ARGOCD_REPO_SERVER_MAX_COMBINED_DIRECTORY_MANIFESTS_SIZE
valueFrom:
configMapKeyRef:
key: reposerver.max.combined.directory.manifests.size
name: argocd-cmd-params-cm
optional: true
- name: ARGOCD_REPO_SERVER_PLUGIN_TAR_EXCLUSIONS
valueFrom:
configMapKeyRef:
key: reposerver.plugin.tar.exclusions
name: argocd-cmd-params-cm
optional: true
- name: HELM_CACHE_HOME
value: /helm-working-dir
- name: HELM_CONFIG_HOME
value: /helm-working-dir
- name: HELM_DATA_HOME
value: /helm-working-dir
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
livenessProbe:
failureThreshold: 3
@@ -10091,7 +10103,7 @@ spec:
- -n
- /usr/local/bin/argocd
- /var/run/argocd/argocd-cmp-server
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
name: copyutil
securityContext:
allowPrivilegeEscalation: false
@@ -10334,7 +10346,7 @@ spec:
key: otlp.address
name: argocd-cmd-params-cm
optional: true
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
livenessProbe:
httpGet:
@@ -10536,7 +10548,7 @@ spec:
key: otlp.address
name: argocd-cmd-params-cm
optional: true
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
livenessProbe:
httpGet:

View File

@@ -616,7 +616,7 @@ spec:
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
name: argocd-applicationset-controller
ports:
@@ -713,7 +713,7 @@ spec:
- -n
- /usr/local/bin/argocd
- /shared/argocd-dex
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
name: copyutil
securityContext:
@@ -753,7 +753,7 @@ spec:
containers:
- command:
- argocd-notifications
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
livenessProbe:
tcpSocket:
@@ -960,13 +960,25 @@ spec:
key: otlp.address
name: argocd-cmd-params-cm
optional: true
- name: ARGOCD_REPO_SERVER_MAX_COMBINED_DIRECTORY_MANIFESTS_SIZE
valueFrom:
configMapKeyRef:
key: reposerver.max.combined.directory.manifests.size
name: argocd-cmd-params-cm
optional: true
- name: ARGOCD_REPO_SERVER_PLUGIN_TAR_EXCLUSIONS
valueFrom:
configMapKeyRef:
key: reposerver.plugin.tar.exclusions
name: argocd-cmd-params-cm
optional: true
- name: HELM_CACHE_HOME
value: /helm-working-dir
- name: HELM_CONFIG_HOME
value: /helm-working-dir
- name: HELM_DATA_HOME
value: /helm-working-dir
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
livenessProbe:
failureThreshold: 3
@@ -1015,7 +1027,7 @@ spec:
- -n
- /usr/local/bin/argocd
- /var/run/argocd/argocd-cmp-server
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
name: copyutil
securityContext:
allowPrivilegeEscalation: false
@@ -1258,7 +1270,7 @@ spec:
key: otlp.address
name: argocd-cmd-params-cm
optional: true
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
livenessProbe:
httpGet:
@@ -1460,7 +1472,7 @@ spec:
key: otlp.address
name: argocd-cmd-params-cm
optional: true
image: quay.io/argoproj/argocd:v2.4.0
image: quay.io/argoproj/argocd:v2.4.4
imagePullPolicy: Always
livenessProbe:
httpGet:

View File

@@ -129,6 +129,7 @@ nav:
- user-guide/compare-options.md
- user-guide/sync-options.md
- user-guide/parameters.md
- user-guide/environment-variables.md
- user-guide/build-environment.md
- user-guide/tracking_strategies.md
- user-guide/resource_tracking.md

View File

@@ -100,7 +100,11 @@ func (c *client) executeRequest(fullMethodName string, msg []byte, md metadata.M
}
func (c *client) startGRPCProxy() (*grpc.Server, net.Listener, error) {
serverAddr := fmt.Sprintf("%s/argocd-%s.sock", os.TempDir(), rand.RandString(16))
randSuffix, err := rand.String(16)
if err != nil {
return nil, nil, fmt.Errorf("failed to generate random socket filename: %w", err)
}
serverAddr := fmt.Sprintf("%s/argocd-%s.sock", os.TempDir(), randSuffix)
ln, err := net.Listen("unix", serverAddr)
if err != nil {

View File

@@ -1542,6 +1542,19 @@ func isValidAction(action string) bool {
return false
}
// TODO: same as validActions, refacotor to use rbacpolicy.ResourceApplications etc.
var validResources = map[string]bool{
"applications": true,
"repositories": true,
"clusters": true,
"exec": true,
"logs": true,
}
func isValidResource(resource string) bool {
return validResources[resource]
}
func validatePolicy(proj string, role string, policy string) error {
policyComponents := strings.Split(policy, ",")
if len(policyComponents) != 6 || strings.Trim(policyComponents[0], " ") != "p" {
@@ -1555,7 +1568,7 @@ func validatePolicy(proj string, role string, policy string) error {
}
// resource
resource := strings.Trim(policyComponents[2], " ")
if resource != "applications" && resource != "repositories" && resource != "clusters" {
if !isValidResource(resource) {
return status.Errorf(codes.InvalidArgument, "invalid policy rule '%s': project resource must be: 'applications', 'repositories' or 'clusters', not '%s'", policy, resource)
}
// action

View File

@@ -2546,10 +2546,19 @@ func Test_validatePolicy_projIsNotRegex(t *testing.T) {
}
func Test_validatePolicy_ValidResource(t *testing.T) {
err := validatePolicy("some-project", "org-admin", "p, proj:some-project:org-admin, repositories, *, some-project/*, allow")
err := validatePolicy("some-project", "org-admin", "p, proj:some-project:org-admin, applications, *, some-project/*, allow")
assert.NoError(t, err)
err = validatePolicy("some-project", "org-admin", "p, proj:some-project:org-admin, repositories, *, some-project/*, allow")
assert.NoError(t, err)
err = validatePolicy("some-project", "org-admin", "p, proj:some-project:org-admin, clusters, *, some-project/*, allow")
assert.NoError(t, err)
err = validatePolicy("some-project", "org-admin", "p, proj:some-project:org-admin, exec, *, some-project/*, allow")
assert.NoError(t, err)
err = validatePolicy("some-project", "org-admin", "p, proj:some-project:org-admin, logs, *, some-project/*, allow")
assert.NoError(t, err)
err = validatePolicy("some-project", "org-admin", "p, proj:some-project:org-admin, unknown, *, some-project/*, allow")
assert.Error(t, err)
}
func TestEnvsubst(t *testing.T) {

View File

@@ -18,6 +18,10 @@ import (
"strings"
"time"
kubeyaml "k8s.io/apimachinery/pkg/util/yaml"
"k8s.io/apimachinery/pkg/api/resource"
"github.com/argoproj/argo-cd/v2/util/io/files"
"github.com/Masterminds/semver/v3"
@@ -71,6 +75,8 @@ const (
ociPrefix = "oci://"
)
var ErrExceededMaxCombinedManifestFileSize = errors.New("exceeded max combined manifest file size")
// Service implements ManifestService interface
type Service struct {
gitCredsStore git.CredsStore
@@ -96,6 +102,8 @@ type RepoServerInitConstants struct {
PauseGenerationOnFailureForMinutes int
PauseGenerationOnFailureForRequests int
SubmoduleEnabled bool
MaxCombinedDirectoryManifestsSize resource.Quantity
CMPTarExcludedGlobs []string
}
// NewService returns a new instance of the Manifest service
@@ -206,7 +214,7 @@ func (s *Service) ListApps(ctx context.Context, q *apiclient.ListAppsRequest) (*
}
defer io.Close(closer)
apps, err := discovery.Discover(ctx, gitClient.Root(), q.EnabledSourceTypes)
apps, err := discovery.Discover(ctx, gitClient.Root(), q.EnabledSourceTypes, s.initConstants.CMPTarExcludedGlobs)
if err != nil {
return nil, err
}
@@ -450,10 +458,15 @@ func (s *Service) runManifestGenAsync(ctx context.Context, repoRoot, commitSHA,
close(ch.errCh)
close(ch.responseCh)
}()
// GenerateManifests mutates the source (applies overrides). Those overrides shouldn't be reflected in the cache
// key. Overrides will break the cache anyway, because changes to overrides will change the revision.
appSourceCopy := q.ApplicationSource.DeepCopy()
var manifestGenResult *apiclient.ManifestResponse
opContext, err := opContextSrc()
if err == nil {
manifestGenResult, err = GenerateManifests(ctx, opContext.appPath, repoRoot, commitSHA, q, false, s.gitCredsStore, WithCMPTarDoneChannel(ch.tarDoneCh))
manifestGenResult, err = GenerateManifests(ctx, opContext.appPath, repoRoot, commitSHA, q, false, s.gitCredsStore, s.initConstants.MaxCombinedDirectoryManifestsSize, WithCMPTarDoneChannel(ch.tarDoneCh), WithCMPTarExcludedGlobs(s.initConstants.CMPTarExcludedGlobs))
}
if err != nil {
// If manifest generation error caching is enabled
@@ -462,9 +475,9 @@ func (s *Service) runManifestGenAsync(ctx context.Context, repoRoot, commitSHA,
// Retrieve a new copy (if available) of the cached response: this ensures we are updating the latest copy of the cache,
// rather than a copy of the cache that occurred before (a potentially lengthy) manifest generation.
innerRes := &cache.CachedManifestResponse{}
cacheErr := s.cache.GetManifests(cacheKey, q.ApplicationSource, q, q.Namespace, q.TrackingMethod, q.AppLabelKey, q.AppName, innerRes)
cacheErr := s.cache.GetManifests(cacheKey, appSourceCopy, q, q.Namespace, q.TrackingMethod, q.AppLabelKey, q.AppName, innerRes)
if cacheErr != nil && cacheErr != reposervercache.ErrCacheMiss {
log.Warnf("manifest cache set error %s: %v", q.ApplicationSource.String(), cacheErr)
log.Warnf("manifest cache set error %s: %v", appSourceCopy.String(), cacheErr)
ch.errCh <- cacheErr
return
}
@@ -478,9 +491,9 @@ func (s *Service) runManifestGenAsync(ctx context.Context, repoRoot, commitSHA,
// Update the cache to include failure information
innerRes.NumberOfConsecutiveFailures++
innerRes.MostRecentError = err.Error()
cacheErr = s.cache.SetManifests(cacheKey, q.ApplicationSource, q, q.Namespace, q.TrackingMethod, q.AppLabelKey, q.AppName, innerRes)
cacheErr = s.cache.SetManifests(cacheKey, appSourceCopy, q, q.Namespace, q.TrackingMethod, q.AppLabelKey, q.AppName, innerRes)
if cacheErr != nil {
log.Warnf("manifest cache set error %s: %v", q.ApplicationSource.String(), cacheErr)
log.Warnf("manifest cache set error %s: %v", appSourceCopy.String(), cacheErr)
ch.errCh <- cacheErr
return
}
@@ -499,9 +512,9 @@ func (s *Service) runManifestGenAsync(ctx context.Context, repoRoot, commitSHA,
}
manifestGenResult.Revision = commitSHA
manifestGenResult.VerifyResult = opContext.verificationResult
err = s.cache.SetManifests(cacheKey, q.ApplicationSource, q, q.Namespace, q.TrackingMethod, q.AppLabelKey, q.AppName, &manifestGenCacheEntry)
err = s.cache.SetManifests(cacheKey, appSourceCopy, q, q.Namespace, q.TrackingMethod, q.AppLabelKey, q.AppName, &manifestGenCacheEntry)
if err != nil {
log.Warnf("manifest cache set error %s/%s: %v", q.ApplicationSource.String(), cacheKey, err)
log.Warnf("manifest cache set error %s/%s: %v", appSourceCopy.String(), cacheKey, err)
}
ch.responseCh <- manifestGenCacheEntry.ManifestResponse
}
@@ -846,7 +859,8 @@ func getRepoCredential(repoCredentials []*v1alpha1.RepoCreds, repoURL string) *v
type GenerateManifestOpt func(*generateManifestOpt)
type generateManifestOpt struct {
cmpTarDoneCh chan<- bool
cmpTarDoneCh chan<- bool
cmpTarExcludedGlobs []string
}
func newGenerateManifestOpt(opts ...GenerateManifestOpt) *generateManifestOpt {
@@ -866,14 +880,23 @@ func WithCMPTarDoneChannel(ch chan<- bool) GenerateManifestOpt {
}
}
// GenerateManifests generates manifests from a path
func GenerateManifests(ctx context.Context, appPath, repoRoot, revision string, q *apiclient.ManifestRequest, isLocal bool, gitCredsStore git.CredsStore, opts ...GenerateManifestOpt) (*apiclient.ManifestResponse, error) {
// WithCMPTarExcludedGlobs defines globs for files to filter out when streaming the tarball
// to a CMP sidecar.
func WithCMPTarExcludedGlobs(excludedGlobs []string) GenerateManifestOpt {
return func(o *generateManifestOpt) {
o.cmpTarExcludedGlobs = excludedGlobs
}
}
// GenerateManifests generates manifests from a path. Overrides are applied as a side effect on the given ApplicationSource.
func GenerateManifests(ctx context.Context, appPath, repoRoot, revision string, q *apiclient.ManifestRequest, isLocal bool, gitCredsStore git.CredsStore, maxCombinedManifestQuantity resource.Quantity, opts ...GenerateManifestOpt) (*apiclient.ManifestResponse, error) {
opt := newGenerateManifestOpt(opts...)
var targetObjs []*unstructured.Unstructured
var dest *v1alpha1.ApplicationDestination
resourceTracking := argo.NewResourceTracking()
appSourceType, err := GetAppSourceType(ctx, q.ApplicationSource, appPath, q.AppName, q.EnabledSourceTypes)
appSourceType, err := GetAppSourceType(ctx, q.ApplicationSource, appPath, q.AppName, q.EnabledSourceTypes, opt.cmpTarExcludedGlobs)
if err != nil {
return nil, err
}
@@ -897,7 +920,7 @@ func GenerateManifests(ctx context.Context, appPath, repoRoot, revision string,
if q.ApplicationSource.Plugin != nil && q.ApplicationSource.Plugin.Name != "" {
targetObjs, err = runConfigManagementPlugin(appPath, repoRoot, env, q, q.Repo.GetGitCreds(gitCredsStore))
} else {
targetObjs, err = runConfigManagementPluginSidecars(ctx, appPath, repoRoot, env, q, q.Repo.GetGitCreds(gitCredsStore), opt.cmpTarDoneCh)
targetObjs, err = runConfigManagementPluginSidecars(ctx, appPath, repoRoot, env, q, q.Repo.GetGitCreds(gitCredsStore), opt.cmpTarDoneCh, opt.cmpTarExcludedGlobs)
if err != nil {
err = fmt.Errorf("plugin sidecar failed. %s", err.Error())
}
@@ -908,7 +931,7 @@ func GenerateManifests(ctx context.Context, appPath, repoRoot, revision string,
directory = &v1alpha1.ApplicationSourceDirectory{}
}
logCtx := log.WithField("application", q.AppName)
targetObjs, err = findManifests(logCtx, appPath, repoRoot, env, *directory, q.EnabledSourceTypes)
targetObjs, err = findManifests(logCtx, appPath, repoRoot, env, *directory, q.EnabledSourceTypes, maxCombinedManifestQuantity)
}
if err != nil {
return nil, err
@@ -985,7 +1008,7 @@ func mergeSourceParameters(source *v1alpha1.ApplicationSource, path, appName str
overrides = append(overrides, filepath.Join(path, fmt.Sprintf(appSourceFile, appName)))
}
var merged v1alpha1.ApplicationSource = *source.DeepCopy()
var merged = *source.DeepCopy()
for _, filename := range overrides {
info, err := os.Stat(filename)
@@ -1031,7 +1054,7 @@ func mergeSourceParameters(source *v1alpha1.ApplicationSource, path, appName str
}
// GetAppSourceType returns explicit application source type or examines a directory and determines its application source type
func GetAppSourceType(ctx context.Context, source *v1alpha1.ApplicationSource, path, appName string, enableGenerateManifests map[string]bool) (v1alpha1.ApplicationSourceType, error) {
func GetAppSourceType(ctx context.Context, source *v1alpha1.ApplicationSource, path, appName string, enableGenerateManifests map[string]bool, tarExcludedGlobs []string) (v1alpha1.ApplicationSourceType, error) {
err := mergeSourceParameters(source, path, appName)
if err != nil {
return "", fmt.Errorf("error while parsing source parameters: %v", err)
@@ -1048,7 +1071,7 @@ func GetAppSourceType(ctx context.Context, source *v1alpha1.ApplicationSource, p
}
return *appSourceType, nil
}
appType, err := discovery.AppType(ctx, path, enableGenerateManifests)
appType, err := discovery.AppType(ctx, path, enableGenerateManifests, tarExcludedGlobs)
if err != nil {
return "", err
}
@@ -1078,63 +1101,30 @@ func isNullList(obj *unstructured.Unstructured) bool {
var manifestFile = regexp.MustCompile(`^.*\.(yaml|yml|json|jsonnet)$`)
// findManifests looks at all yaml files in a directory and unmarshals them into a list of unstructured objects
func findManifests(logCtx *log.Entry, appPath string, repoRoot string, env *v1alpha1.Env, directory v1alpha1.ApplicationSourceDirectory, enabledManifestGeneration map[string]bool) ([]*unstructured.Unstructured, error) {
func findManifests(logCtx *log.Entry, appPath string, repoRoot string, env *v1alpha1.Env, directory v1alpha1.ApplicationSourceDirectory, enabledManifestGeneration map[string]bool, maxCombinedManifestQuantity resource.Quantity) ([]*unstructured.Unstructured, error) {
// Validate the directory before loading any manifests to save memory.
potentiallyValidManifests, err := getPotentiallyValidManifests(logCtx, appPath, repoRoot, directory.Recurse, directory.Include, directory.Exclude, maxCombinedManifestQuantity)
if err != nil {
logCtx.Errorf("failed to get potentially valid manifests: %s", err)
return nil, fmt.Errorf("failed to get potentially valid manifests: %w", err)
}
var objs []*unstructured.Unstructured
err := filepath.Walk(appPath, func(path string, f os.FileInfo, err error) error {
if err != nil {
return err
}
relPath, err := filepath.Rel(appPath, path)
if err != nil {
return fmt.Errorf("failed to get relative path of symlink: %w", err)
}
if files.IsSymlink(f) {
realPath, err := filepath.EvalSymlinks(path)
if err != nil {
logCtx.Debugf("error checking symlink realpath: %s", err)
if os.IsNotExist(err) {
log.Warnf("ignoring out-of-bounds symlink at %q: %s", relPath, err)
return nil
} else {
return fmt.Errorf("failed to evaluate symlink at %q: %w", relPath, err)
}
}
if !files.Inbound(realPath, appPath) {
logCtx.Warnf("illegal filepath in symlink: %s", realPath)
return fmt.Errorf("illegal filepath in symlink at %q", relPath)
}
}
if f.IsDir() {
if path != appPath && !directory.Recurse {
return filepath.SkipDir
} else {
return nil
}
}
for _, potentiallyValidManifest := range potentiallyValidManifests {
manifestPath := potentiallyValidManifest.path
manifestFileInfo := potentiallyValidManifest.fileInfo
if !manifestFile.MatchString(f.Name()) {
return nil
}
if directory.Exclude != "" && glob.Match(directory.Exclude, relPath) {
return nil
}
if directory.Include != "" && !glob.Match(directory.Include, relPath) {
return nil
}
if strings.HasSuffix(f.Name(), ".jsonnet") {
if strings.HasSuffix(manifestFileInfo.Name(), ".jsonnet") {
if !discovery.IsManifestGenerationEnabled(v1alpha1.ApplicationSourceTypeDirectory, enabledManifestGeneration) {
return nil
continue
}
vm, err := makeJsonnetVm(appPath, repoRoot, directory.Jsonnet, env)
if err != nil {
return err
return nil, err
}
jsonStr, err := vm.EvaluateFile(path)
jsonStr, err := vm.EvaluateFile(manifestPath)
if err != nil {
return status.Errorf(codes.FailedPrecondition, "Failed to evaluate jsonnet %q: %v", f.Name(), err)
return nil, status.Errorf(codes.FailedPrecondition, "Failed to evaluate jsonnet %q: %v", manifestFileInfo.Name(), err)
}
// attempt to unmarshal either array or single object
@@ -1146,49 +1136,207 @@ func findManifests(logCtx *log.Entry, appPath string, repoRoot string, env *v1al
var jsonObj unstructured.Unstructured
err = json.Unmarshal([]byte(jsonStr), &jsonObj)
if err != nil {
return status.Errorf(codes.FailedPrecondition, "Failed to unmarshal generated json %q: %v", f.Name(), err)
return nil, status.Errorf(codes.FailedPrecondition, "Failed to unmarshal generated json %q: %v", manifestFileInfo.Name(), err)
}
objs = append(objs, &jsonObj)
}
} else {
out, err := utfutil.ReadFile(path, utfutil.UTF8)
err := getObjsFromYAMLOrJson(logCtx, manifestPath, manifestFileInfo.Name(), &objs)
if err != nil {
return err
}
if strings.HasSuffix(f.Name(), ".json") {
var obj unstructured.Unstructured
err = json.Unmarshal(out, &obj)
if err != nil {
return status.Errorf(codes.FailedPrecondition, "Failed to unmarshal %q: %v", f.Name(), err)
}
objs = append(objs, &obj)
} else {
yamlObjs, err := kube.SplitYAML(out)
if err != nil {
if len(yamlObjs) > 0 {
// If we get here, we had a multiple objects in a single YAML file which had some
// valid k8s objects, but errors parsing others (within the same file). It's very
// likely the user messed up a portion of the YAML, so report on that.
return status.Errorf(codes.FailedPrecondition, "Failed to unmarshal %q: %v", f.Name(), err)
}
// Otherwise, let's see if it looks like a resource, if yes, we return error
if bytes.Contains(out, []byte("apiVersion:")) &&
bytes.Contains(out, []byte("kind:")) &&
bytes.Contains(out, []byte("metadata:")) {
return status.Errorf(codes.FailedPrecondition, "Failed to unmarshal %q: %v", f.Name(), err)
}
// Otherwise, it might be a unrelated YAML file which we will ignore
return nil
}
objs = append(objs, yamlObjs...)
return nil, err
}
}
}
return objs, nil
}
// getObjsFromYAMLOrJson unmarshals the given yaml or json file and appends it to the given list of objects.
func getObjsFromYAMLOrJson(logCtx *log.Entry, manifestPath string, filename string, objs *[]*unstructured.Unstructured) error {
reader, err := utfutil.OpenFile(manifestPath, utfutil.UTF8)
if err != nil {
return status.Errorf(codes.FailedPrecondition, "Failed to open %q", manifestPath)
}
defer func() {
err := reader.Close()
if err != nil {
logCtx.Errorf("failed to close %q - potential memory leak", manifestPath)
}
}()
if strings.HasSuffix(filename, ".json") {
var obj unstructured.Unstructured
decoder := json.NewDecoder(reader)
err = decoder.Decode(&obj)
if err != nil {
return status.Errorf(codes.FailedPrecondition, "Failed to unmarshal %q: %v", filename, err)
}
if decoder.More() {
return status.Errorf(codes.FailedPrecondition, "Found multiple objects in %q. Only single objects are allowed in JSON files.", filename)
}
*objs = append(*objs, &obj)
} else {
yamlObjs, err := splitYAMLOrJSON(reader)
if err != nil {
if len(yamlObjs) > 0 {
// If we get here, we had a multiple objects in a single YAML file which had some
// valid k8s objects, but errors parsing others (within the same file). It's very
// likely the user messed up a portion of the YAML, so report on that.
return status.Errorf(codes.FailedPrecondition, "Failed to unmarshal %q: %v", filename, err)
}
// Read the whole file to check whether it looks like a manifest.
out, err := utfutil.ReadFile(manifestPath, utfutil.UTF8)
// Otherwise, let's see if it looks like a resource, if yes, we return error
if bytes.Contains(out, []byte("apiVersion:")) &&
bytes.Contains(out, []byte("kind:")) &&
bytes.Contains(out, []byte("metadata:")) {
return status.Errorf(codes.FailedPrecondition, "Failed to unmarshal %q: %v", filename, err)
}
// Otherwise, it might be an unrelated YAML file which we will ignore
}
*objs = append(*objs, yamlObjs...)
}
return nil
}
// splitYAMLOrJSON reads a YAML or JSON file and gets each document as an unstructured object. If the unmarshaller
// encounters an error, objects read up until the error are returned.
func splitYAMLOrJSON(reader goio.Reader) ([]*unstructured.Unstructured, error) {
d := kubeyaml.NewYAMLOrJSONDecoder(reader, 4096)
var objs []*unstructured.Unstructured
for {
u := &unstructured.Unstructured{}
if err := d.Decode(&u); err != nil {
if err == goio.EOF {
break
}
return objs, fmt.Errorf("failed to unmarshal manifest: %v", err)
}
if u == nil {
continue
}
objs = append(objs, u)
}
return objs, nil
}
// getPotentiallyValidManifestFile checks whether the given path/FileInfo may be a valid manifest file. Returns a non-nil error if
// there was an error that should not be handled by ignoring the file. Returns non-nil realFileInfo if the file is a
// potential manifest. Returns a non-empty ignoreMessage if there's a message that should be logged about why the file
// was skipped. If realFileInfo is nil and the ignoreMessage is empty, there's no need to log the ignoreMessage; the
// file was skipped for a mundane reason.
//
// The file is still only a "potentially" valid manifest file because it could be invalid JSON or YAML, or it might not
// be a valid Kubernetes resource. This function tests everything possible without actually reading the file.
//
// repoPath must be absolute.
func getPotentiallyValidManifestFile(path string, f os.FileInfo, appPath, repoRoot, include, exclude string) (realFileInfo os.FileInfo, warning string, err error) {
relPath, err := filepath.Rel(appPath, path)
if err != nil {
return nil, "", fmt.Errorf("failed to get relative path of %q: %w", path, err)
}
if !manifestFile.MatchString(f.Name()) {
return nil, "", nil
}
// If the file is a symlink, these will be overridden with the destination file's info.
var relRealPath = relPath
realFileInfo = f
if files.IsSymlink(f) {
realPath, err := filepath.EvalSymlinks(path)
if err != nil {
if os.IsNotExist(err) {
return nil, fmt.Sprintf("destination of symlink %q is missing", relPath), nil
}
return nil, "", fmt.Errorf("failed to evaluate symlink at %q: %w", relPath, err)
}
if !files.Inbound(realPath, repoRoot) {
return nil, "", fmt.Errorf("illegal filepath in symlink at %q", relPath)
}
realFileInfo, err = os.Stat(realPath)
if err != nil {
if os.IsNotExist(err) {
// This should have been caught by filepath.EvalSymlinks, but check again since that function's docs
// don't promise to return this error.
return nil, fmt.Sprintf("destination of symlink %q is missing at %q", relPath, realPath), nil
}
return nil, "", fmt.Errorf("failed to get file info for symlink at %q to %q: %w", relPath, realPath, err)
}
relRealPath, err = filepath.Rel(repoRoot, realPath)
if err != nil {
return nil, "", fmt.Errorf("failed to get relative path of %q: %w", realPath, err)
}
}
// FileInfo.Size() behavior is platform-specific for non-regular files. Allow only regular files, so we guarantee
// accurate file sizes.
if !realFileInfo.Mode().IsRegular() {
return nil, fmt.Sprintf("ignoring symlink at %q to non-regular file %q", relPath, relRealPath), nil
}
if exclude != "" && glob.Match(exclude, relPath) {
return nil, "", nil
}
if include != "" && !glob.Match(include, relPath) {
return nil, "", nil
}
return realFileInfo, "", nil
}
type potentiallyValidManifest struct {
path string
fileInfo os.FileInfo
}
// getPotentiallyValidManifests ensures that 1) there are no errors while checking for potential manifest files in the given dir
// and 2) the combined file size of the potentially-valid manifest files does not exceed the limit.
func getPotentiallyValidManifests(logCtx *log.Entry, appPath string, repoRoot string, recurse bool, include string, exclude string, maxCombinedManifestQuantity resource.Quantity) ([]potentiallyValidManifest, error) {
maxCombinedManifestFileSize := maxCombinedManifestQuantity.Value()
var currentCombinedManifestFileSize = int64(0)
var potentiallyValidManifests []potentiallyValidManifest
err := filepath.Walk(appPath, func(path string, f os.FileInfo, err error) error {
if err != nil {
return err
}
if f.IsDir() {
if path != appPath && !recurse {
return filepath.SkipDir
}
return nil
}
realFileInfo, warning, err := getPotentiallyValidManifestFile(path, f, appPath, repoRoot, include, exclude)
if err != nil {
return fmt.Errorf("invalid manifest file %q: %w", path, err)
}
if realFileInfo == nil {
if warning != "" {
logCtx.Warnf("skipping manifest file %q: %s", path, warning)
}
return nil
}
// Don't count jsonnet file size against max. It's jsonnet's responsibility to manage memory usage.
if !strings.HasSuffix(f.Name(), ".jsonnet") {
// We use the realFileInfo size (which is guaranteed to be a regular file instead of a symlink or other
// non-regular file) because .Size() behavior is platform-specific for non-regular files.
currentCombinedManifestFileSize += realFileInfo.Size()
if maxCombinedManifestFileSize != 0 && currentCombinedManifestFileSize > maxCombinedManifestFileSize {
return ErrExceededMaxCombinedManifestFileSize
}
}
potentiallyValidManifests = append(potentiallyValidManifests, potentiallyValidManifest{path: path, fileInfo: f})
return nil
})
if err != nil {
// Not wrapping, because this error should be wrapped by the caller.
return nil, err
}
return objs, nil
return potentiallyValidManifests, nil
}
func makeJsonnetVm(appPath string, repoRoot string, sourceJsonnet v1alpha1.ApplicationSourceJsonnet, env *v1alpha1.Env) (*jsonnet.VM, error) {
@@ -1327,7 +1475,7 @@ func getPluginEnvs(envVars *v1alpha1.Env, q *apiclient.ManifestRequest, creds gi
return env, nil
}
func runConfigManagementPluginSidecars(ctx context.Context, appPath, repoPath string, envVars *v1alpha1.Env, q *apiclient.ManifestRequest, creds git.Creds, tarDoneCh chan<- bool) ([]*unstructured.Unstructured, error) {
func runConfigManagementPluginSidecars(ctx context.Context, appPath, repoPath string, envVars *v1alpha1.Env, q *apiclient.ManifestRequest, creds git.Creds, tarDoneCh chan<- bool, tarExcludedGlobs []string) ([]*unstructured.Unstructured, error) {
// compute variables.
env, err := getPluginEnvs(envVars, q, creds, true)
if err != nil {
@@ -1335,14 +1483,14 @@ func runConfigManagementPluginSidecars(ctx context.Context, appPath, repoPath st
}
// detect config management plugin server (sidecar)
conn, cmpClient, err := discovery.DetectConfigManagementPlugin(ctx, appPath, env)
conn, cmpClient, err := discovery.DetectConfigManagementPlugin(ctx, appPath, env, tarExcludedGlobs)
if err != nil {
return nil, err
}
defer io.Close(conn)
// generate manifests using commands provided in plugin config file in detected cmp-server sidecar
cmpManifests, err := generateManifestsCMP(ctx, appPath, repoPath, env, cmpClient, tarDoneCh)
cmpManifests, err := generateManifestsCMP(ctx, appPath, repoPath, env, cmpClient, tarDoneCh, tarExcludedGlobs)
if err != nil {
return nil, fmt.Errorf("error generating manifests in cmp: %s", err)
}
@@ -1360,7 +1508,7 @@ func runConfigManagementPluginSidecars(ctx context.Context, appPath, repoPath st
// generateManifestsCMP will send the appPath files to the cmp-server over a gRPC stream.
// The cmp-server will generate the manifests. Returns a response object with the generated
// manifests.
func generateManifestsCMP(ctx context.Context, appPath, repoPath string, env []string, cmpClient pluginclient.ConfigManagementPluginServiceClient, tarDoneCh chan<- bool) (*pluginclient.ManifestResponse, error) {
func generateManifestsCMP(ctx context.Context, appPath, repoPath string, env []string, cmpClient pluginclient.ConfigManagementPluginServiceClient, tarDoneCh chan<- bool, tarExcludedGlobs []string) (*pluginclient.ManifestResponse, error) {
generateManifestStream, err := cmpClient.GenerateManifest(ctx, grpc_retry.Disable())
if err != nil {
return nil, fmt.Errorf("error getting generateManifestStream: %s", err)
@@ -1368,7 +1516,7 @@ func generateManifestsCMP(ctx context.Context, appPath, repoPath string, env []s
opts := []cmp.SenderOption{
cmp.WithTarDoneChan(tarDoneCh),
}
err = cmp.SendRepoStream(generateManifestStream.Context(), appPath, repoPath, generateManifestStream, env, opts...)
err = cmp.SendRepoStream(generateManifestStream.Context(), appPath, repoPath, generateManifestStream, env, tarExcludedGlobs, opts...)
if err != nil {
return nil, fmt.Errorf("error sending file to cmp-server: %s", err)
}
@@ -1386,7 +1534,7 @@ func (s *Service) GetAppDetails(ctx context.Context, q *apiclient.RepoServerAppD
return err
}
appSourceType, err := GetAppSourceType(ctx, q.Source, opContext.appPath, q.AppName, q.EnabledSourceTypes)
appSourceType, err := GetAppSourceType(ctx, q.Source, opContext.appPath, q.AppName, q.EnabledSourceTypes, s.initConstants.CMPTarExcludedGlobs)
if err != nil {
return err
}
@@ -1461,8 +1609,12 @@ func populateHelmAppDetails(res *apiclient.RepoAppDetailsResponse, appPath strin
return err
}
if err := loadFileIntoIfExists(filepath.Join(appPath, "values.yaml"), &res.Helm.Values); err != nil {
return err
if resolvedValuesPath, _, err := pathutil.ResolveFilePath(appPath, repoRoot, "values.yaml", []string{}); err == nil {
if err := loadFileIntoIfExists(resolvedValuesPath, &res.Helm.Values); err != nil {
return err
}
} else {
log.Warnf("Values file %s is not allowed: %v", filepath.Join(appPath, "values.yaml"), err)
}
var resolvedSelectedValueFiles []pathutil.ResolvedFilePath
// drop not allowed values files
@@ -1470,10 +1622,10 @@ func populateHelmAppDetails(res *apiclient.RepoAppDetailsResponse, appPath strin
if resolvedFile, _, err := pathutil.ResolveFilePath(appPath, repoRoot, file, q.GetValuesFileSchemes()); err == nil {
resolvedSelectedValueFiles = append(resolvedSelectedValueFiles, resolvedFile)
} else {
log.Debugf("Values file %s is not allowed: %v", file, err)
log.Warnf("Values file %s is not allowed: %v", file, err)
}
}
params, err := h.GetParameters(resolvedSelectedValueFiles)
params, err := h.GetParameters(resolvedSelectedValueFiles, appPath, repoRoot)
if err != nil {
return err
}
@@ -1492,15 +1644,16 @@ func populateHelmAppDetails(res *apiclient.RepoAppDetailsResponse, appPath strin
return nil
}
func loadFileIntoIfExists(path string, destination *string) error {
info, err := os.Stat(path)
func loadFileIntoIfExists(path pathutil.ResolvedFilePath, destination *string) error {
stringPath := string(path)
info, err := os.Stat(stringPath)
if err == nil && !info.IsDir() {
if bytes, err := ioutil.ReadFile(path); err != nil {
*destination = string(bytes)
} else {
bytes, err := ioutil.ReadFile(stringPath)
if err != nil {
return err
}
*destination = string(bytes)
}
return nil

View File

@@ -5,8 +5,8 @@ import (
"encoding/json"
"errors"
"fmt"
log "github.com/sirupsen/logrus"
goio "io"
"io/fs"
"io/ioutil"
"os"
"os/exec"
@@ -17,6 +17,9 @@ import (
"testing"
"time"
log "github.com/sirupsen/logrus"
"k8s.io/apimachinery/pkg/api/resource"
"github.com/ghodss/yaml"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock"
@@ -130,6 +133,31 @@ func newServiceWithCommitSHA(root, revision string) *Service {
return service
}
// createSymlink creates a symlink with name linkName to file destName in
// workingDir
func createSymlink(t *testing.T, workingDir, destName, linkName string) error {
oldWorkingDir, err := os.Getwd()
if err != nil {
return err
}
if workingDir != "" {
err = os.Chdir(workingDir)
if err != nil {
return err
}
defer func() {
if err := os.Chdir(oldWorkingDir); err != nil {
t.Fatal(err.Error())
}
}()
}
err = os.Symlink(destName, linkName)
if err != nil {
return err
}
return nil
}
func TestGenerateYamlManifestInDir(t *testing.T) {
service := newService("../..")
@@ -145,39 +173,39 @@ func TestGenerateYamlManifestInDir(t *testing.T) {
assert.Equal(t, countOfManifests, len(res1.Manifests))
// this will test concatenated manifests to verify we split YAMLs correctly
res2, err := GenerateManifests(context.Background(), "./testdata/concatenated", "/", "", &q, false, &git.NoopCredsStore{})
res2, err := GenerateManifests(context.Background(), "./testdata/concatenated", "/", "", &q, false, &git.NoopCredsStore{}, resource.MustParse("0"))
assert.NoError(t, err)
assert.Equal(t, 3, len(res2.Manifests))
}
func Test_GenerateManifests_NoOutOfBoundsAccess(t *testing.T) {
testCases := []struct{
name string
outOfBoundsFilename string
testCases := []struct {
name string
outOfBoundsFilename string
outOfBoundsFileContents string
mustNotContain string // Optional string that must not appear in error or manifest output. If empty, use outOfBoundsFileContents.
mustNotContain string // Optional string that must not appear in error or manifest output. If empty, use outOfBoundsFileContents.
}{
{
name: "out of bounds JSON file should not appear in error output",
outOfBoundsFilename: "test.json",
name: "out of bounds JSON file should not appear in error output",
outOfBoundsFilename: "test.json",
outOfBoundsFileContents: `{"some": "json"}`,
},
{
name: "malformed JSON file contents should not appear in error output",
outOfBoundsFilename: "test.json",
name: "malformed JSON file contents should not appear in error output",
outOfBoundsFilename: "test.json",
outOfBoundsFileContents: "$",
},
{
name: "out of bounds JSON manifest should not appear in manifest output",
name: "out of bounds JSON manifest should not appear in manifest output",
outOfBoundsFilename: "test.json",
// JSON marshalling is deterministic. So if there's a leak, exactly this should appear in the manifests.
outOfBoundsFileContents: `{"apiVersion":"v1","kind":"Secret","metadata":{"name":"test","namespace":"default"},"type":"Opaque"}`,
},
{
name: "out of bounds YAML manifest should not appear in manifest output",
outOfBoundsFilename: "test.yaml",
name: "out of bounds YAML manifest should not appear in manifest output",
outOfBoundsFilename: "test.yaml",
outOfBoundsFileContents: "apiVersion: v1\nkind: Secret\nmetadata:\n name: test\n namespace: default\ntype: Opaque",
mustNotContain: `{"apiVersion":"v1","kind":"Secret","metadata":{"name":"test","namespace":"default"},"type":"Opaque"}`,
mustNotContain: `{"apiVersion":"v1","kind":"Secret","metadata":{"name":"test","namespace":"default"},"type":"Opaque"}`,
},
}
@@ -201,7 +229,7 @@ func Test_GenerateManifests_NoOutOfBoundsAccess(t *testing.T) {
}
q := apiclient.ManifestRequest{Repo: &argoappv1.Repository{}, ApplicationSource: &argoappv1.ApplicationSource{}}
res, err := GenerateManifests(context.Background(), repoDir, "", "", &q, false, &git.NoopCredsStore{})
res, err := GenerateManifests(context.Background(), repoDir, "", "", &q, false, &git.NoopCredsStore{}, resource.MustParse("0"))
require.Error(t, err)
assert.NotContains(t, err.Error(), mustNotContain)
assert.Contains(t, err.Error(), "illegal filepath")
@@ -216,7 +244,7 @@ func TestGenerateManifests_MissingSymlinkDestination(t *testing.T) {
require.NoError(t, err)
q := apiclient.ManifestRequest{Repo: &argoappv1.Repository{}, ApplicationSource: &argoappv1.ApplicationSource{}}
_, err = GenerateManifests(context.Background(), repoDir, "", "", &q, false, &git.NoopCredsStore{})
_, err = GenerateManifests(context.Background(), repoDir, "", "", &q, false, &git.NoopCredsStore{}, resource.MustParse("0"))
require.NoError(t, err)
}
@@ -1046,15 +1074,15 @@ func TestGenerateNullList(t *testing.T) {
}
func TestIdentifyAppSourceTypeByAppDirWithKustomizations(t *testing.T) {
sourceType, err := GetAppSourceType(context.Background(), &argoappv1.ApplicationSource{}, "./testdata/kustomization_yaml", "testapp", map[string]bool{})
sourceType, err := GetAppSourceType(context.Background(), &argoappv1.ApplicationSource{}, "./testdata/kustomization_yaml", "testapp", map[string]bool{}, []string{})
assert.Nil(t, err)
assert.Equal(t, argoappv1.ApplicationSourceTypeKustomize, sourceType)
sourceType, err = GetAppSourceType(context.Background(), &argoappv1.ApplicationSource{}, "./testdata/kustomization_yml", "testapp", map[string]bool{})
sourceType, err = GetAppSourceType(context.Background(), &argoappv1.ApplicationSource{}, "./testdata/kustomization_yml", "testapp", map[string]bool{}, []string{})
assert.Nil(t, err)
assert.Equal(t, argoappv1.ApplicationSourceTypeKustomize, sourceType)
sourceType, err = GetAppSourceType(context.Background(), &argoappv1.ApplicationSource{}, "./testdata/Kustomization", "testapp", map[string]bool{})
sourceType, err = GetAppSourceType(context.Background(), &argoappv1.ApplicationSource{}, "./testdata/Kustomization", "testapp", map[string]bool{}, []string{})
assert.Nil(t, err)
assert.Equal(t, argoappv1.ApplicationSourceTypeKustomize, sourceType)
}
@@ -1107,7 +1135,7 @@ func TestGenerateFromUTF16(t *testing.T) {
Repo: &argoappv1.Repository{},
ApplicationSource: &argoappv1.ApplicationSource{},
}
res1, err := GenerateManifests(context.Background(), "./testdata/utf-16", "/", "", &q, false, &git.NoopCredsStore{})
res1, err := GenerateManifests(context.Background(), "./testdata/utf-16", "/", "", &q, false, &git.NoopCredsStore{}, resource.MustParse("0"))
assert.Nil(t, err)
assert.Equal(t, 2, len(res1.Manifests))
}
@@ -1119,17 +1147,20 @@ func TestListApps(t *testing.T) {
assert.NoError(t, err)
expectedApps := map[string]string{
"Kustomization": "Kustomize",
"app-parameters/multi": "Kustomize",
"app-parameters/single-app-only": "Kustomize",
"app-parameters/single-global": "Kustomize",
"invalid-helm": "Helm",
"invalid-kustomize": "Kustomize",
"kustomization_yaml": "Kustomize",
"kustomization_yml": "Kustomize",
"my-chart": "Helm",
"my-chart-2": "Helm",
"values-files": "Helm",
"Kustomization": "Kustomize",
"app-parameters/multi": "Kustomize",
"app-parameters/single-app-only": "Kustomize",
"app-parameters/single-global": "Kustomize",
"app-parameters/single-global-helm": "Helm",
"invalid-helm": "Helm",
"in-bounds-values-file-link": "Helm",
"invalid-kustomize": "Kustomize",
"kustomization_yaml": "Kustomize",
"kustomization_yml": "Kustomize",
"my-chart": "Helm",
"my-chart-2": "Helm",
"out-of-bounds-values-file-link": "Helm",
"values-files": "Helm",
}
assert.Equal(t, expectedApps, res.Apps)
}
@@ -1448,6 +1479,7 @@ func mkTempParameters(source string) string {
func runWithTempTestdata(t *testing.T, path string, runner func(t *testing.T, path string)) {
tempDir := mkTempParameters("./testdata/app-parameters")
runner(t, filepath.Join(tempDir, "app-parameters", path))
os.RemoveAll(tempDir)
}
func TestGenerateManifestsWithAppParameterFile(t *testing.T) {
@@ -1480,6 +1512,35 @@ func TestGenerateManifestsWithAppParameterFile(t *testing.T) {
})
})
t.Run("Single global override Helm", func(t *testing.T) {
runWithTempTestdata(t, "single-global-helm", func(t *testing.T, path string) {
service := newService(".")
manifests, err := service.GenerateManifest(context.Background(), &apiclient.ManifestRequest{
Repo: &argoappv1.Repository{},
ApplicationSource: &argoappv1.ApplicationSource{
Path: path,
},
})
require.NoError(t, err)
resourceByKindName := make(map[string]*unstructured.Unstructured)
for _, manifest := range manifests.Manifests {
var un unstructured.Unstructured
err := yaml.Unmarshal([]byte(manifest), &un)
if !assert.NoError(t, err) {
return
}
resourceByKindName[fmt.Sprintf("%s/%s", un.GetKind(), un.GetName())] = &un
}
deployment, ok := resourceByKindName["Deployment/guestbook-ui"]
require.True(t, ok)
containers, ok, _ := unstructured.NestedSlice(deployment.Object, "spec", "template", "spec", "containers")
require.True(t, ok)
image, ok, _ := unstructured.NestedString(containers[0].(map[string]interface{}), "image")
require.True(t, ok)
assert.Equal(t, "gcr.io/heptio-images/ks-guestbook-demo:0.2", image)
})
})
t.Run("Application specific override", func(t *testing.T) {
service := newService(".")
runWithTempTestdata(t, "single-app-only", func(t *testing.T, path string) {
@@ -1539,6 +1600,28 @@ func TestGenerateManifestsWithAppParameterFile(t *testing.T) {
assert.Equal(t, "gcr.io/heptio-images/ks-guestbook-demo:0.1", image)
})
})
t.Run("Override info does not appear in cache key", func(t *testing.T) {
service := newService(".")
runWithTempTestdata(t, "single-global", func(t *testing.T, path string) {
source := &argoappv1.ApplicationSource{
Path: path,
}
sourceCopy := source.DeepCopy() // make a copy in case GenerateManifest mutates it.
_, err := service.GenerateManifest(context.Background(), &apiclient.ManifestRequest{
Repo: &argoappv1.Repository{},
ApplicationSource: sourceCopy,
AppName: "test",
})
assert.NoError(t, err)
res := &cache.CachedManifestResponse{}
// Try to pull from the cache with a `source` that does not include any overrides. Overrides should not be
// part of the cache key, because you can't get the overrides without a repo operation. And avoiding repo
// operations is the point of the cache.
err = service.cache.GetManifests(mock.Anything, source, &argoappv1.ClusterInfo{}, "", "", "", "test", res)
assert.NoError(t, err)
})
})
}
func TestGenerateManifestWithAnnotatedAndRegularGitTagHashes(t *testing.T) {
@@ -1653,7 +1736,7 @@ func TestFindResources(t *testing.T) {
Recurse: true,
Include: tc.include,
Exclude: tc.exclude,
}, map[string]bool{})
}, map[string]bool{}, resource.MustParse("0"))
if !assert.NoError(t, err) {
return
}
@@ -1670,7 +1753,7 @@ func TestFindManifests_Exclude(t *testing.T) {
objs, err := findManifests(&log.Entry{}, "testdata/app-include-exclude", ".", nil, argoappv1.ApplicationSourceDirectory{
Recurse: true,
Exclude: "subdir/deploymentSub.yaml",
}, map[string]bool{})
}, map[string]bool{}, resource.MustParse("0"))
if !assert.NoError(t, err) || !assert.Len(t, objs, 1) {
return
@@ -1683,7 +1766,7 @@ func TestFindManifests_Exclude_NothingMatches(t *testing.T) {
objs, err := findManifests(&log.Entry{}, "testdata/app-include-exclude", ".", nil, argoappv1.ApplicationSourceDirectory{
Recurse: true,
Exclude: "nothing.yaml",
}, map[string]bool{})
}, map[string]bool{}, resource.MustParse("0"))
if !assert.NoError(t, err) || !assert.Len(t, objs, 2) {
return
@@ -1693,6 +1776,477 @@ func TestFindManifests_Exclude_NothingMatches(t *testing.T) {
[]string{"nginx-deployment", "nginx-deployment-sub"}, []string{objs[0].GetName(), objs[1].GetName()})
}
func tempDir(t *testing.T) string {
dir, err := ioutil.TempDir(".", "")
require.NoError(t, err)
t.Cleanup(func() {
err = os.RemoveAll(dir)
if err != nil {
panic(err)
}
})
absDir, err := filepath.Abs(dir)
require.NoError(t, err)
return absDir
}
func walkFor(t *testing.T, root string, testPath string, run func(info fs.FileInfo)) {
var hitExpectedPath = false
err := filepath.Walk(root, func(path string, info fs.FileInfo, err error) error {
if path == testPath {
require.NoError(t, err)
hitExpectedPath = true
run(info)
}
return nil
})
require.NoError(t, err)
assert.True(t, hitExpectedPath, "did not hit expected path when walking directory")
}
func Test_getPotentiallyValidManifestFile(t *testing.T) {
// These tests use filepath.Walk instead of os.Stat to get file info, because FileInfo from os.Stat does not return
// true for IsSymlink like os.Walk does.
// These tests do not use t.TempDir() because those directories can contain symlinks which cause test to fail
// InBound checks.
t.Run("non-JSON/YAML is skipped with an empty ignore message", func(t *testing.T) {
appDir := tempDir(t)
filePath := filepath.Join(appDir, "not-json-or-yaml")
file, err := os.OpenFile(filePath, os.O_RDONLY|os.O_CREATE, 0644)
require.NoError(t, err)
err = file.Close()
require.NoError(t, err)
walkFor(t, appDir, filePath, func(info fs.FileInfo) {
realFileInfo, ignoreMessage, err := getPotentiallyValidManifestFile(filePath, info, appDir, appDir, "", "")
assert.Nil(t, realFileInfo)
assert.Empty(t, ignoreMessage)
assert.NoError(t, err)
})
})
t.Run("circular link should throw an error", func(t *testing.T) {
appDir := tempDir(t)
aPath := filepath.Join(appDir, "a.json")
bPath := filepath.Join(appDir, "b.json")
err := os.Symlink(bPath, aPath)
require.NoError(t, err)
err = os.Symlink(aPath, bPath)
require.NoError(t, err)
walkFor(t, appDir, aPath, func(info fs.FileInfo) {
realFileInfo, ignoreMessage, err := getPotentiallyValidManifestFile(aPath, info, appDir, appDir, "", "")
assert.Nil(t, realFileInfo)
assert.Empty(t, ignoreMessage)
assert.ErrorContains(t, err, "too many links")
})
})
t.Run("symlink with missing destination should throw an error", func(t *testing.T) {
appDir := tempDir(t)
aPath := filepath.Join(appDir, "a.json")
bPath := filepath.Join(appDir, "b.json")
err := os.Symlink(bPath, aPath)
require.NoError(t, err)
walkFor(t, appDir, aPath, func(info fs.FileInfo) {
realFileInfo, ignoreMessage, err := getPotentiallyValidManifestFile(aPath, info, appDir, appDir, "", "")
assert.Nil(t, realFileInfo)
assert.NotEmpty(t, ignoreMessage)
assert.NoError(t, err)
})
})
t.Run("out-of-bounds symlink should throw an error", func(t *testing.T) {
appDir := tempDir(t)
linkPath := filepath.Join(appDir, "a.json")
err := os.Symlink("..", linkPath)
require.NoError(t, err)
walkFor(t, appDir, linkPath, func(info fs.FileInfo) {
realFileInfo, ignoreMessage, err := getPotentiallyValidManifestFile(linkPath, info, appDir, appDir, "", "")
assert.Nil(t, realFileInfo)
assert.Empty(t, ignoreMessage)
assert.ErrorContains(t, err, "illegal filepath in symlink")
})
})
t.Run("symlink to a non-regular file should be skipped with warning", func(t *testing.T) {
appDir := tempDir(t)
dirPath := filepath.Join(appDir, "test.dir")
err := os.MkdirAll(dirPath, 0644)
require.NoError(t, err)
linkPath := filepath.Join(appDir, "test.json")
err = os.Symlink(dirPath, linkPath)
require.NoError(t, err)
walkFor(t, appDir, linkPath, func(info fs.FileInfo) {
realFileInfo, ignoreMessage, err := getPotentiallyValidManifestFile(linkPath, info, appDir, appDir, "", "")
assert.Nil(t, realFileInfo)
assert.Contains(t, ignoreMessage, "non-regular file")
assert.NoError(t, err)
})
})
t.Run("non-included file should be skipped with no message", func(t *testing.T) {
appDir := tempDir(t)
filePath := filepath.Join(appDir, "not-included.yaml")
file, err := os.OpenFile(filePath, os.O_RDONLY|os.O_CREATE, 0644)
require.NoError(t, err)
err = file.Close()
require.NoError(t, err)
walkFor(t, appDir, filePath, func(info fs.FileInfo) {
realFileInfo, ignoreMessage, err := getPotentiallyValidManifestFile(filePath, info, appDir, appDir, "*.json", "")
assert.Nil(t, realFileInfo)
assert.Empty(t, ignoreMessage)
assert.NoError(t, err)
})
})
t.Run("excluded file should be skipped with no message", func(t *testing.T) {
appDir := tempDir(t)
filePath := filepath.Join(appDir, "excluded.json")
file, err := os.OpenFile(filePath, os.O_RDONLY|os.O_CREATE, 0644)
require.NoError(t, err)
err = file.Close()
require.NoError(t, err)
walkFor(t, appDir, filePath, func(info fs.FileInfo) {
realFileInfo, ignoreMessage, err := getPotentiallyValidManifestFile(filePath, info, appDir, appDir, "", "excluded.*")
assert.Nil(t, realFileInfo)
assert.Empty(t, ignoreMessage)
assert.NoError(t, err)
})
})
t.Run("symlink to a regular file is potentially valid", func(t *testing.T) {
appDir := tempDir(t)
filePath := filepath.Join(appDir, "regular-file")
file, err := os.OpenFile(filePath, os.O_RDONLY|os.O_CREATE, 0644)
require.NoError(t, err)
err = file.Close()
require.NoError(t, err)
linkPath := filepath.Join(appDir, "link.json")
err = os.Symlink(filePath, linkPath)
require.NoError(t, err)
walkFor(t, appDir, linkPath, func(info fs.FileInfo) {
realFileInfo, ignoreMessage, err := getPotentiallyValidManifestFile(linkPath, info, appDir, appDir, "", "")
assert.NotNil(t, realFileInfo)
assert.Empty(t, ignoreMessage)
assert.NoError(t, err)
})
})
t.Run("a regular file is potentially valid", func(t *testing.T) {
appDir := tempDir(t)
filePath := filepath.Join(appDir, "regular-file.json")
file, err := os.OpenFile(filePath, os.O_RDONLY|os.O_CREATE, 0644)
require.NoError(t, err)
err = file.Close()
require.NoError(t, err)
walkFor(t, appDir, filePath, func(info fs.FileInfo) {
realFileInfo, ignoreMessage, err := getPotentiallyValidManifestFile(filePath, info, appDir, appDir, "", "")
assert.NotNil(t, realFileInfo)
assert.Empty(t, ignoreMessage)
assert.NoError(t, err)
})
})
t.Run("realFileInfo is for the destination rather than the symlink", func(t *testing.T) {
appDir := tempDir(t)
filePath := filepath.Join(appDir, "regular-file")
file, err := os.OpenFile(filePath, os.O_RDONLY|os.O_CREATE, 0644)
require.NoError(t, err)
err = file.Close()
require.NoError(t, err)
linkPath := filepath.Join(appDir, "link.json")
err = os.Symlink(filePath, linkPath)
require.NoError(t, err)
walkFor(t, appDir, linkPath, func(info fs.FileInfo) {
realFileInfo, ignoreMessage, err := getPotentiallyValidManifestFile(linkPath, info, appDir, appDir, "", "")
assert.NotNil(t, realFileInfo)
assert.Equal(t, filepath.Base(filePath), realFileInfo.Name())
assert.Empty(t, ignoreMessage)
assert.NoError(t, err)
})
})
}
func Test_getPotentiallyValidManifests(t *testing.T) {
// Tests which return no manifests and an error check to make sure the directory exists before running. A missing
// directory would produce those same results.
logCtx := log.WithField("test", "test")
t.Run("unreadable file throws error", func(t *testing.T) {
appDir := t.TempDir()
unreadablePath := filepath.Join(appDir, "unreadable.json")
err := os.WriteFile(unreadablePath, []byte{}, 0666)
require.NoError(t, err)
err = os.Chmod(appDir, 0000)
require.NoError(t, err)
manifests, err := getPotentiallyValidManifests(logCtx, appDir, appDir, false, "", "", resource.MustParse("0"))
assert.Empty(t, manifests)
assert.Error(t, err)
// allow cleanup
err = os.Chmod(appDir, 0777)
if err != nil {
panic(err)
}
})
t.Run("no recursion when recursion is disabled", func(t *testing.T) {
manifests, err := getPotentiallyValidManifests(logCtx, "./testdata/recurse", "./testdata/recurse", false, "", "", resource.MustParse("0"))
assert.Len(t, manifests, 1)
assert.NoError(t, err)
})
t.Run("recursion when recursion is enabled", func(t *testing.T) {
manifests, err := getPotentiallyValidManifests(logCtx, "./testdata/recurse", "./testdata/recurse", true, "", "", resource.MustParse("0"))
assert.Len(t, manifests, 2)
assert.NoError(t, err)
})
t.Run("non-JSON/YAML is skipped", func(t *testing.T) {
manifests, err := getPotentiallyValidManifests(logCtx, "./testdata/non-manifest-file", "./testdata/non-manifest-file", false, "", "", resource.MustParse("0"))
assert.Empty(t, manifests)
assert.NoError(t, err)
})
t.Run("circular link should throw an error", func(t *testing.T) {
const testDir = "./testdata/circular-link"
require.DirExists(t, testDir)
require.NoError(t, createSymlink(t, testDir, "a.json", "b.json"))
defer os.Remove(path.Join(testDir, "a.json"))
require.NoError(t, createSymlink(t, testDir, "b.json", "a.json"))
defer os.Remove(path.Join(testDir, "b.json"))
manifests, err := getPotentiallyValidManifests(logCtx, "./testdata/circular-link", "./testdata/circular-link", false, "", "", resource.MustParse("0"))
assert.Empty(t, manifests)
assert.Error(t, err)
})
t.Run("out-of-bounds symlink should throw an error", func(t *testing.T) {
require.DirExists(t, "./testdata/out-of-bounds-link")
manifests, err := getPotentiallyValidManifests(logCtx, "./testdata/out-of-bounds-link", "./testdata/out-of-bounds-link", false, "", "", resource.MustParse("0"))
assert.Empty(t, manifests)
assert.Error(t, err)
})
t.Run("symlink to a regular file works", func(t *testing.T) {
repoRoot, err := filepath.Abs("./testdata/in-bounds-link")
require.NoError(t, err)
appPath, err := filepath.Abs("./testdata/in-bounds-link/app")
require.NoError(t, err)
manifests, err := getPotentiallyValidManifests(logCtx, appPath, repoRoot, false, "", "", resource.MustParse("0"))
assert.Len(t, manifests, 1)
assert.NoError(t, err)
})
t.Run("symlink to nowhere should be ignored", func(t *testing.T) {
manifests, err := getPotentiallyValidManifests(logCtx, "./testdata/link-to-nowhere", "./testdata/link-to-nowhere", false, "", "", resource.MustParse("0"))
assert.Empty(t, manifests)
assert.NoError(t, err)
})
t.Run("link to over-sized manifest fails", func(t *testing.T) {
repoRoot, err := filepath.Abs("./testdata/in-bounds-link")
require.NoError(t, err)
appPath, err := filepath.Abs("./testdata/in-bounds-link/app")
require.NoError(t, err)
// The file is 35 bytes.
manifests, err := getPotentiallyValidManifests(logCtx, appPath, repoRoot, false, "", "", resource.MustParse("34"))
assert.Empty(t, manifests)
assert.ErrorIs(t, err, ErrExceededMaxCombinedManifestFileSize)
})
t.Run("group of files should be limited at precisely the sum of their size", func(t *testing.T) {
// There is a total of 10 files, ech file being 10 bytes.
manifests, err := getPotentiallyValidManifests(logCtx, "./testdata/several-files", "./testdata/several-files", false, "", "", resource.MustParse("365"))
assert.Len(t, manifests, 10)
assert.NoError(t, err)
manifests, err = getPotentiallyValidManifests(logCtx, "./testdata/several-files", "./testdata/several-files", false, "", "", resource.MustParse("100"))
assert.Empty(t, manifests)
assert.ErrorIs(t, err, ErrExceededMaxCombinedManifestFileSize)
})
}
func Test_findManifests(t *testing.T) {
logCtx := log.WithField("test", "test")
noRecurse := argoappv1.ApplicationSourceDirectory{Recurse: false}
t.Run("unreadable file throws error", func(t *testing.T) {
appDir := t.TempDir()
unreadablePath := filepath.Join(appDir, "unreadable.json")
err := os.WriteFile(unreadablePath, []byte{}, 0666)
require.NoError(t, err)
err = os.Chmod(appDir, 0000)
require.NoError(t, err)
manifests, err := findManifests(logCtx, appDir, appDir, nil, noRecurse, nil, resource.MustParse("0"))
assert.Empty(t, manifests)
assert.Error(t, err)
// allow cleanup
err = os.Chmod(appDir, 0777)
if err != nil {
panic(err)
}
})
t.Run("no recursion when recursion is disabled", func(t *testing.T) {
manifests, err := findManifests(logCtx, "./testdata/recurse", "./testdata/recurse", nil, noRecurse, nil, resource.MustParse("0"))
assert.Len(t, manifests, 2)
assert.NoError(t, err)
})
t.Run("recursion when recursion is enabled", func(t *testing.T) {
recurse := argoappv1.ApplicationSourceDirectory{Recurse: true}
manifests, err := findManifests(logCtx, "./testdata/recurse", "./testdata/recurse", nil, recurse, nil, resource.MustParse("0"))
assert.Len(t, manifests, 4)
assert.NoError(t, err)
})
t.Run("non-JSON/YAML is skipped", func(t *testing.T) {
manifests, err := findManifests(logCtx, "./testdata/non-manifest-file", "./testdata/non-manifest-file", nil, noRecurse, nil, resource.MustParse("0"))
assert.Empty(t, manifests)
assert.NoError(t, err)
})
t.Run("circular link should throw an error", func(t *testing.T) {
const testDir = "./testdata/circular-link"
require.DirExists(t, testDir)
require.NoError(t, createSymlink(t, testDir, "a.json", "b.json"))
defer os.Remove(path.Join(testDir, "a.json"))
require.NoError(t, createSymlink(t, testDir, "b.json", "a.json"))
defer os.Remove(path.Join(testDir, "b.json"))
manifests, err := findManifests(logCtx, "./testdata/circular-link", "./testdata/circular-link", nil, noRecurse, nil, resource.MustParse("0"))
assert.Empty(t, manifests)
assert.Error(t, err)
})
t.Run("out-of-bounds symlink should throw an error", func(t *testing.T) {
require.DirExists(t, "./testdata/out-of-bounds-link")
manifests, err := findManifests(logCtx, "./testdata/out-of-bounds-link", "./testdata/out-of-bounds-link", nil, noRecurse, nil, resource.MustParse("0"))
assert.Empty(t, manifests)
assert.Error(t, err)
})
t.Run("symlink to a regular file works", func(t *testing.T) {
repoRoot, err := filepath.Abs("./testdata/in-bounds-link")
require.NoError(t, err)
appPath, err := filepath.Abs("./testdata/in-bounds-link/app")
require.NoError(t, err)
manifests, err := findManifests(logCtx, appPath, repoRoot, nil, noRecurse, nil, resource.MustParse("0"))
assert.Len(t, manifests, 1)
assert.NoError(t, err)
})
t.Run("symlink to nowhere should be ignored", func(t *testing.T) {
manifests, err := findManifests(logCtx, "./testdata/link-to-nowhere", "./testdata/link-to-nowhere", nil, noRecurse, nil, resource.MustParse("0"))
assert.Empty(t, manifests)
assert.NoError(t, err)
})
t.Run("link to over-sized manifest fails", func(t *testing.T) {
repoRoot, err := filepath.Abs("./testdata/in-bounds-link")
require.NoError(t, err)
appPath, err := filepath.Abs("./testdata/in-bounds-link/app")
require.NoError(t, err)
// The file is 35 bytes.
manifests, err := findManifests(logCtx, appPath, repoRoot, nil, noRecurse, nil, resource.MustParse("34"))
assert.Empty(t, manifests)
assert.ErrorIs(t, err, ErrExceededMaxCombinedManifestFileSize)
})
t.Run("group of files should be limited at precisely the sum of their size", func(t *testing.T) {
// There is a total of 10 files, each file being 10 bytes.
manifests, err := findManifests(logCtx, "./testdata/several-files", "./testdata/several-files", nil, noRecurse, nil, resource.MustParse("365"))
assert.Len(t, manifests, 10)
assert.NoError(t, err)
manifests, err = findManifests(logCtx, "./testdata/several-files", "./testdata/several-files", nil, noRecurse, nil, resource.MustParse("364"))
assert.Empty(t, manifests)
assert.ErrorIs(t, err, ErrExceededMaxCombinedManifestFileSize)
})
t.Run("jsonnet isn't counted against size limit", func(t *testing.T) {
// Each file is 36 bytes. Only the 36-byte json file should be counted against the limit.
manifests, err := findManifests(logCtx, "./testdata/jsonnet-and-json", "./testdata/jsonnet-and-json", nil, noRecurse, nil, resource.MustParse("36"))
assert.Len(t, manifests, 2)
assert.NoError(t, err)
manifests, err = findManifests(logCtx, "./testdata/jsonnet-and-json", "./testdata/jsonnet-and-json", nil, noRecurse, nil, resource.MustParse("35"))
assert.Empty(t, manifests)
assert.ErrorIs(t, err, ErrExceededMaxCombinedManifestFileSize)
})
t.Run("partially valid YAML file throws an error", func(t *testing.T) {
require.DirExists(t, "./testdata/partially-valid-yaml")
manifests, err := findManifests(logCtx, "./testdata/partially-valid-yaml", "./testdata/partially-valid-yaml", nil, noRecurse, nil, resource.MustParse("0"))
assert.Empty(t, manifests)
assert.Error(t, err)
})
t.Run("invalid manifest throws an error", func(t *testing.T) {
require.DirExists(t, "./testdata/invalid-manifests")
manifests, err := findManifests(logCtx, "./testdata/invalid-manifests", "./testdata/invalid-manifests", nil, noRecurse, nil, resource.MustParse("0"))
assert.Empty(t, manifests)
assert.Error(t, err)
})
t.Run("irrelevant YAML gets skipped, relevant YAML gets parsed", func(t *testing.T) {
manifests, err := findManifests(logCtx, "./testdata/irrelevant-yaml", "./testdata/irrelevant-yaml", nil, noRecurse, nil, resource.MustParse("0"))
assert.Len(t, manifests, 1)
assert.NoError(t, err)
})
t.Run("multiple JSON objects in one file throws an error", func(t *testing.T) {
require.DirExists(t, "./testdata/json-list")
manifests, err := findManifests(logCtx, "./testdata/json-list", "./testdata/json-list", nil, noRecurse, nil, resource.MustParse("0"))
assert.Empty(t, manifests)
assert.Error(t, err)
})
t.Run("invalid JSON throws an error", func(t *testing.T) {
require.DirExists(t, "./testdata/invalid-json")
manifests, err := findManifests(logCtx, "./testdata/invalid-json", "./testdata/invalid-json", nil, noRecurse, nil, resource.MustParse("0"))
assert.Empty(t, manifests)
assert.Error(t, err)
})
t.Run("valid JSON returns manifest and no error", func(t *testing.T) {
manifests, err := findManifests(logCtx, "./testdata/valid-json", "./testdata/valid-json", nil, noRecurse, nil, resource.MustParse("0"))
assert.Len(t, manifests, 1)
assert.NoError(t, err)
})
t.Run("YAML with an empty document doesn't throw an error", func(t *testing.T) {
manifests, err := findManifests(logCtx, "./testdata/yaml-with-empty-document", "./testdata/yaml-with-empty-document", nil, noRecurse, nil, resource.MustParse("0"))
assert.Len(t, manifests, 1)
assert.NoError(t, err)
})
}
func TestTestRepoOCI(t *testing.T) {
service := newService(".")
_, err := service.TestRepository(context.Background(), &apiclient.TestRepositoryRequest{
@@ -1904,3 +2458,23 @@ func Test_populateHelmAppDetails(t *testing.T) {
assert.Len(t, res.Helm.Parameters, 3)
assert.Len(t, res.Helm.ValueFiles, 4)
}
func Test_populateHelmAppDetails_values_symlinks(t *testing.T) {
t.Run("inbound", func(t *testing.T) {
res := apiclient.RepoAppDetailsResponse{}
q := apiclient.RepoServerAppDetailsQuery{Repo: &argoappv1.Repository{}, Source: &argoappv1.ApplicationSource{}}
err := populateHelmAppDetails(&res, "./testdata/in-bounds-values-file-link/", "./testdata/in-bounds-values-file-link/", &q)
require.NoError(t, err)
assert.NotEmpty(t, res.Helm.Values)
assert.NotEmpty(t, res.Helm.Parameters)
})
t.Run("out of bounds", func(t *testing.T) {
res := apiclient.RepoAppDetailsResponse{}
q := apiclient.RepoServerAppDetailsQuery{Repo: &argoappv1.Repository{}, Source: &argoappv1.ApplicationSource{}}
err := populateHelmAppDetails(&res, "./testdata/out-of-bounds-values-file-link/", "./testdata/out-of-bounds-values-file-link/", &q)
require.NoError(t, err)
assert.Empty(t, res.Helm.Values)
assert.Empty(t, res.Helm.Parameters)
})
}

View File

@@ -0,0 +1,4 @@
helm:
parameters:
- name: image.tag
value: '0.2'

View File

@@ -0,0 +1,2 @@
name: my-chart
version: 1.1.0

View File

@@ -0,0 +1,18 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: guestbook-ui
spec:
selector:
matchLabels:
app: guestbook-ui
template:
metadata:
labels:
app: guestbook-ui
spec:
containers:
- image: gcr.io/heptio-images/ks-guestbook-demo:{{.Values.image.tag}}
name: guestbook-ui
ports:
- containerPort: 81

View File

@@ -0,0 +1,2 @@
image:
tag: 0.1

View File

View File

@@ -0,0 +1 @@
../cm.yaml

View File

@@ -0,0 +1,2 @@
apiVersion: v1
kind: ServiceAccount

View File

@@ -0,0 +1,2 @@
name: my-chart
version: 1.1.0

View File

@@ -0,0 +1 @@
some: yaml

View File

@@ -0,0 +1 @@
values-2.yaml

View File

@@ -0,0 +1 @@
[

View File

@@ -0,0 +1 @@
some: [irrelevant, yaml]

View File

@@ -0,0 +1,2 @@
apiVersion: v1
kind: ConfigMap

View File

@@ -0,0 +1,2 @@
{"apiVersion": "v1", "kind": "ConfigMap"}
{"apiVersion": "v1", "kind": "ConfigMap"}

View File

@@ -0,0 +1 @@
{"apiVersion": "v1", "kind": "Pod"}

View File

@@ -0,0 +1 @@
{"apiVersion": "v1", "kind": "Pod"}

View File

@@ -0,0 +1 @@
nowhere

View File

@@ -0,0 +1 @@
../out-of-bounds.json

View File

@@ -0,0 +1,2 @@
name: my-chart
version: 1.1.0

View File

@@ -0,0 +1 @@
../out-of-bounds.yaml

View File

View File

@@ -0,0 +1 @@
some: yaml

View File

@@ -0,0 +1,4 @@
apiVersion: v1
kind: ConfigMap
---
invalid:

View File

@@ -0,0 +1 @@
{"apiVersion": "v1", "kind": "ConfigMap"}

View File

@@ -0,0 +1,2 @@
apiVersion: v1
kind: ConfigMap

View File

@@ -0,0 +1 @@
{"apiVersion": "v1", "kind": "ConfigMap"}

View File

@@ -0,0 +1,2 @@
apiVersion: v1
kind: ConfigMap

View File

@@ -0,0 +1 @@
{"apiVersion": "v1", "kind": "ConfigMap"}

View File

@@ -0,0 +1,2 @@
apiVersion: v1
kind: ConfigMap

View File

@@ -0,0 +1 @@
{"apiVersion": "v1", "kind": "ConfigMap"}

View File

@@ -0,0 +1,2 @@
apiVersion: v1
kind: ConfigMap

View File

@@ -0,0 +1 @@
{"apiVersion": "v1", "kind": "ConfigMap"}

View File

@@ -0,0 +1,2 @@
apiVersion: v1
kind: ConfigMap

View File

@@ -0,0 +1 @@
This file shouldn't be counted in the manifest file size limit, because it isn't JSON or YAML.

View File

@@ -0,0 +1 @@
{"apiVersion": "v1", "kind": "ConfigMap"}

View File

@@ -0,0 +1,4 @@
apiVersion: v1
kind: ConfigMap
---
---

View File

@@ -132,7 +132,7 @@ var backoff = wait.Backoff{
var (
clientConstraint = fmt.Sprintf(">= %s", common.MinClientVersion)
baseHRefRegex = regexp.MustCompile(`<base href="(.*)">`)
baseHRefRegex = regexp.MustCompile(`<base href="(.*?)">`)
// limits number of concurrent login requests to prevent password brute forcing. If set to 0 then no limit is enforced.
maxConcurrentLoginRequestsCount = 50
replicasCount = 1
@@ -282,11 +282,97 @@ func (a *ArgoCDServer) healthCheck(r *http.Request) error {
return nil
}
type Listeners struct {
Main net.Listener
Metrics net.Listener
GatewayConn *grpc.ClientConn
}
func (l *Listeners) Close() error {
if l.Main != nil {
if err := l.Main.Close(); err != nil {
return err
}
l.Main = nil
}
if l.Metrics != nil {
if err := l.Metrics.Close(); err != nil {
return err
}
l.Metrics = nil
}
if l.GatewayConn != nil {
if err := l.GatewayConn.Close(); err != nil {
return err
}
l.GatewayConn = nil
}
return nil
}
func startListener(host string, port int) (net.Listener, error) {
var conn net.Listener
var realErr error
_ = wait.ExponentialBackoff(backoff, func() (bool, error) {
conn, realErr = net.Listen("tcp", fmt.Sprintf("%s:%d", host, port))
if realErr != nil {
return false, nil
}
return true, nil
})
return conn, realErr
}
func (a *ArgoCDServer) Listen() (*Listeners, error) {
mainLn, err := startListener(a.ListenHost, a.ListenPort)
if err != nil {
return nil, err
}
metricsLn, err := startListener(a.ListenHost, a.MetricsPort)
if err != nil {
io.Close(mainLn)
return nil, err
}
var dOpts []grpc.DialOption
dOpts = append(dOpts, grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(apiclient.MaxGRPCMessageSize)))
dOpts = append(dOpts, grpc.WithUserAgent(fmt.Sprintf("%s/%s", common.ArgoCDUserAgentName, common.GetVersion().Version)))
dOpts = append(dOpts, grpc.WithUnaryInterceptor(otelgrpc.UnaryClientInterceptor()))
dOpts = append(dOpts, grpc.WithStreamInterceptor(otelgrpc.StreamClientInterceptor()))
if a.useTLS() {
// The following sets up the dial Options for grpc-gateway to talk to gRPC server over TLS.
// grpc-gateway is just translating HTTP/HTTPS requests as gRPC requests over localhost,
// so we need to supply the same certificates to establish the connections that a normal,
// external gRPC client would need.
tlsConfig := a.settings.TLSConfig()
if a.TLSConfigCustomizer != nil {
a.TLSConfigCustomizer(tlsConfig)
}
tlsConfig.InsecureSkipVerify = true
dCreds := credentials.NewTLS(tlsConfig)
dOpts = append(dOpts, grpc.WithTransportCredentials(dCreds))
} else {
dOpts = append(dOpts, grpc.WithTransportCredentials(insecure.NewCredentials()))
}
conn, err := grpc.Dial(fmt.Sprintf("localhost:%d", a.ListenPort), dOpts...)
if err != nil {
io.Close(mainLn)
io.Close(metricsLn)
return nil, err
}
return &Listeners{Main: mainLn, Metrics: metricsLn, GatewayConn: conn}, nil
}
// Init starts informers used by the API server
func (a *ArgoCDServer) Init(ctx context.Context) {
go a.projInformer.Run(ctx.Done())
go a.appInformer.Run(ctx.Done())
}
// Run runs the API Server
// We use k8s.io/code-generator/cmd/go-to-protobuf to generate the .proto files from the API types.
// k8s.io/ go-to-protobuf uses protoc-gen-gogo, which comes from gogo/protobuf (a fork of
// golang/protobuf).
func (a *ArgoCDServer) Run(ctx context.Context, port int, metricsPort int) {
func (a *ArgoCDServer) Run(ctx context.Context, listeners *Listeners) {
a.userStateStorage.Init(ctx)
grpcS, appResourceTreeFn := a.newGRPCServer()
@@ -294,10 +380,10 @@ func (a *ArgoCDServer) Run(ctx context.Context, port int, metricsPort int) {
var httpS *http.Server
var httpsS *http.Server
if a.useTLS() {
httpS = newRedirectServer(port, a.RootPath)
httpsS = a.newHTTPServer(ctx, port, grpcWebS, appResourceTreeFn)
httpS = newRedirectServer(a.ListenPort, a.RootPath)
httpsS = a.newHTTPServer(ctx, a.ListenPort, grpcWebS, appResourceTreeFn, listeners.GatewayConn)
} else {
httpS = a.newHTTPServer(ctx, port, grpcWebS, appResourceTreeFn)
httpS = a.newHTTPServer(ctx, a.ListenPort, grpcWebS, appResourceTreeFn, listeners.GatewayConn)
}
if a.RootPath != "" {
httpS.Handler = withRootPath(httpS.Handler, a)
@@ -311,26 +397,13 @@ func (a *ArgoCDServer) Run(ctx context.Context, port int, metricsPort int) {
httpsS.Handler = &bug21955Workaround{handler: httpsS.Handler}
}
metricsServ := metrics.NewMetricsServer(a.ListenHost, metricsPort)
metricsServ := metrics.NewMetricsServer(a.ListenHost, a.MetricsPort)
if a.RedisClient != nil {
cacheutil.CollectMetrics(a.RedisClient, metricsServ)
}
// Start listener
var conn net.Listener
var realErr error
_ = wait.ExponentialBackoff(backoff, func() (bool, error) {
conn, realErr = net.Listen("tcp", fmt.Sprintf("%s:%d", a.ListenHost, port))
if realErr != nil {
a.log.Warnf("failed listen: %v", realErr)
return false, nil
}
return true, nil
})
errors.CheckError(realErr)
// CMux is used to support servicing gRPC and HTTP1.1+JSON on the same port
tcpm := cmux.New(conn)
tcpm := cmux.New(listeners.Main)
var tlsm cmux.CMux
var grpcL net.Listener
var httpL net.Listener
@@ -360,10 +433,7 @@ func (a *ArgoCDServer) Run(ctx context.Context, port int, metricsPort int) {
// Start the muxed listeners for our servers
log.Infof("argocd %s serving on port %d (url: %s, tls: %v, namespace: %s, sso: %v)",
common.GetVersion(), port, a.settings.URL, a.useTLS(), a.Namespace, a.settings.IsSSOConfigured())
go a.projInformer.Run(ctx.Done())
go a.appInformer.Run(ctx.Done())
common.GetVersion(), a.ListenPort, a.settings.URL, a.useTLS(), a.Namespace, a.settings.IsSSOConfigured())
go func() { a.checkServeErr("grpcS", grpcS.Serve(grpcL)) }()
go func() { a.checkServeErr("httpS", httpS.Serve(httpL)) }()
@@ -374,17 +444,13 @@ func (a *ArgoCDServer) Run(ctx context.Context, port int, metricsPort int) {
go a.watchSettings()
go a.rbacPolicyLoader(ctx)
go func() { a.checkServeErr("tcpm", tcpm.Serve()) }()
go func() { a.checkServeErr("metrics", metricsServ.ListenAndServe()) }()
go func() { a.checkServeErr("metrics", metricsServ.Serve(listeners.Metrics)) }()
if !cache.WaitForCacheSync(ctx.Done(), a.projInformer.HasSynced, a.appInformer.HasSynced) {
log.Fatal("Timed out waiting for project cache to sync")
}
a.stopCh = make(chan struct{})
<-a.stopCh
errors.CheckError(conn.Close())
if err := metricsServ.Shutdown(ctx); err != nil {
log.Fatalf("Failed to gracefully shutdown metrics server: %v", err)
}
}
func (a *ArgoCDServer) Initialized() bool {
@@ -702,7 +768,7 @@ func compressHandler(handler http.Handler) http.Handler {
// newHTTPServer returns the HTTP server to serve HTTP/HTTPS requests. This is implemented
// using grpc-gateway as a proxy to the gRPC server.
func (a *ArgoCDServer) newHTTPServer(ctx context.Context, port int, grpcWebHandler http.Handler, appResourceTreeFn application.AppResourceTreeFn) *http.Server {
func (a *ArgoCDServer) newHTTPServer(ctx context.Context, port int, grpcWebHandler http.Handler, appResourceTreeFn application.AppResourceTreeFn, conn *grpc.ClientConn) *http.Server {
endpoint := fmt.Sprintf("localhost:%d", port)
mux := http.NewServeMux()
httpS := http.Server{
@@ -718,26 +784,6 @@ func (a *ArgoCDServer) newHTTPServer(ctx context.Context, port int, grpcWebHandl
},
},
}
var dOpts []grpc.DialOption
dOpts = append(dOpts, grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(apiclient.MaxGRPCMessageSize)))
dOpts = append(dOpts, grpc.WithUserAgent(fmt.Sprintf("%s/%s", common.ArgoCDUserAgentName, common.GetVersion().Version)))
dOpts = append(dOpts, grpc.WithUnaryInterceptor(otelgrpc.UnaryClientInterceptor()))
dOpts = append(dOpts, grpc.WithStreamInterceptor(otelgrpc.StreamClientInterceptor()))
if a.useTLS() {
// The following sets up the dial Options for grpc-gateway to talk to gRPC server over TLS.
// grpc-gateway is just translating HTTP/HTTPS requests as gRPC requests over localhost,
// so we need to supply the same certificates to establish the connections that a normal,
// external gRPC client would need.
tlsConfig := a.settings.TLSConfig()
if a.TLSConfigCustomizer != nil {
a.TLSConfigCustomizer(tlsConfig)
}
tlsConfig.InsecureSkipVerify = true
dCreds := credentials.NewTLS(tlsConfig)
dOpts = append(dOpts, grpc.WithTransportCredentials(dCreds))
} else {
dOpts = append(dOpts, grpc.WithTransportCredentials(insecure.NewCredentials()))
}
// HTTP 1.1+JSON Server
// grpc-ecosystem/grpc-gateway is used to proxy HTTP requests to the corresponding gRPC call
@@ -790,17 +836,17 @@ func (a *ArgoCDServer) newHTTPServer(ctx context.Context, port int, grpcWebHandl
terminalHandler.ServeHTTP(writer, request)
})
mustRegisterGWHandler(versionpkg.RegisterVersionServiceHandlerFromEndpoint, ctx, gwmux, endpoint, dOpts)
mustRegisterGWHandler(clusterpkg.RegisterClusterServiceHandlerFromEndpoint, ctx, gwmux, endpoint, dOpts)
mustRegisterGWHandler(applicationpkg.RegisterApplicationServiceHandlerFromEndpoint, ctx, gwmux, endpoint, dOpts)
mustRegisterGWHandler(repositorypkg.RegisterRepositoryServiceHandlerFromEndpoint, ctx, gwmux, endpoint, dOpts)
mustRegisterGWHandler(repocredspkg.RegisterRepoCredsServiceHandlerFromEndpoint, ctx, gwmux, endpoint, dOpts)
mustRegisterGWHandler(sessionpkg.RegisterSessionServiceHandlerFromEndpoint, ctx, gwmux, endpoint, dOpts)
mustRegisterGWHandler(settingspkg.RegisterSettingsServiceHandlerFromEndpoint, ctx, gwmux, endpoint, dOpts)
mustRegisterGWHandler(projectpkg.RegisterProjectServiceHandlerFromEndpoint, ctx, gwmux, endpoint, dOpts)
mustRegisterGWHandler(accountpkg.RegisterAccountServiceHandlerFromEndpoint, ctx, gwmux, endpoint, dOpts)
mustRegisterGWHandler(certificatepkg.RegisterCertificateServiceHandlerFromEndpoint, ctx, gwmux, endpoint, dOpts)
mustRegisterGWHandler(gpgkeypkg.RegisterGPGKeyServiceHandlerFromEndpoint, ctx, gwmux, endpoint, dOpts)
mustRegisterGWHandler(versionpkg.RegisterVersionServiceHandler, ctx, gwmux, conn)
mustRegisterGWHandler(clusterpkg.RegisterClusterServiceHandler, ctx, gwmux, conn)
mustRegisterGWHandler(applicationpkg.RegisterApplicationServiceHandler, ctx, gwmux, conn)
mustRegisterGWHandler(repositorypkg.RegisterRepositoryServiceHandler, ctx, gwmux, conn)
mustRegisterGWHandler(repocredspkg.RegisterRepoCredsServiceHandler, ctx, gwmux, conn)
mustRegisterGWHandler(sessionpkg.RegisterSessionServiceHandler, ctx, gwmux, conn)
mustRegisterGWHandler(settingspkg.RegisterSettingsServiceHandler, ctx, gwmux, conn)
mustRegisterGWHandler(projectpkg.RegisterProjectServiceHandler, ctx, gwmux, conn)
mustRegisterGWHandler(accountpkg.RegisterAccountServiceHandler, ctx, gwmux, conn)
mustRegisterGWHandler(certificatepkg.RegisterCertificateServiceHandler, ctx, gwmux, conn)
mustRegisterGWHandler(gpgkeypkg.RegisterGPGKeyServiceHandler, ctx, gwmux, conn)
// Swagger UI
swagger.ServeSwaggerUI(mux, assets.SwaggerJSON, "/swagger-ui", a.RootPath)
@@ -893,7 +939,7 @@ func (s *ArgoCDServer) getIndexData() ([]byte, error) {
if s.BaseHRef == "/" || s.BaseHRef == "" {
s.indexData = data
} else {
s.indexData = []byte(baseHRefRegex.ReplaceAllString(string(data), fmt.Sprintf(`<base href="/%s/">`, strings.Trim(s.BaseHRef, "/"))))
s.indexData = []byte(replaceBaseHRef(string(data), fmt.Sprintf(`<base href="/%s/">`, strings.Trim(s.BaseHRef, "/"))))
}
})
@@ -968,16 +1014,20 @@ func isMainJsBundle(url *url.URL) bool {
return mainJsBundleRegex.Match([]byte(filename))
}
type registerFunc func(ctx context.Context, mux *runtime.ServeMux, endpoint string, opts []grpc.DialOption) error
type registerFunc func(ctx context.Context, mux *runtime.ServeMux, conn *grpc.ClientConn) error
// mustRegisterGWHandler is a convenience function to register a gateway handler
func mustRegisterGWHandler(register registerFunc, ctx context.Context, mux *runtime.ServeMux, endpoint string, opts []grpc.DialOption) {
err := register(ctx, mux, endpoint, opts)
func mustRegisterGWHandler(register registerFunc, ctx context.Context, mux *runtime.ServeMux, conn *grpc.ClientConn) {
err := register(ctx, mux, conn)
if err != nil {
panic(err)
}
}
func replaceBaseHRef(data string, replaceWith string) string {
return baseHRefRegex.ReplaceAllString(data, replaceWith)
}
// Authenticate checks for the presence of a valid token when accessing server-side resources.
func (a *ArgoCDServer) Authenticate(ctx context.Context) (context.Context, error) {
if a.DisableAuth {
@@ -1010,6 +1060,7 @@ func (a *ArgoCDServer) Authenticate(ctx context.Context) (context.Context, error
if !argoCDSettings.AnonymousUserEnabled {
return ctx, claimsErr
} else {
// nolint:staticcheck
ctx = context.WithValue(ctx, "claims", "")
}
}

View File

@@ -29,20 +29,17 @@ func TestUserAgent(t *testing.T) {
s, closer := fakeServer()
defer closer()
lns, err := s.Listen()
assert.NoError(t, err)
cancelInformer := test.StartInformer(s.projInformer)
defer cancelInformer()
port, err := test.GetFreePort()
assert.NoError(t, err)
metricsPort, err := test.GetFreePort()
assert.NoError(t, err)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go s.Run(ctx, port, metricsPort)
s.Init(ctx)
go s.Run(ctx, lns)
defer func() { time.Sleep(3 * time.Second) }()
err = test.WaitForPortListen(fmt.Sprintf("127.0.0.1:%d", port), 10*time.Second)
assert.NoError(t, err)
type testData struct {
userAgent string
errorMsg string
@@ -72,7 +69,7 @@ func TestUserAgent(t *testing.T) {
for _, test := range tests {
opts := apiclient.ClientOptions{
ServerAddr: fmt.Sprintf("localhost:%d", port),
ServerAddr: fmt.Sprintf("localhost:%d", s.ListenPort),
PlainText: true,
UserAgent: test.userAgent,
}
@@ -99,25 +96,21 @@ func Test_StaticHeaders(t *testing.T) {
{
s, closer := fakeServer()
defer closer()
lns, err := s.Listen()
assert.NoError(t, err)
cancelInformer := test.StartInformer(s.projInformer)
defer cancelInformer()
port, err := test.GetFreePort()
assert.NoError(t, err)
metricsPort, err := test.GetFreePort()
assert.NoError(t, err)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go s.Run(ctx, port, metricsPort)
s.Init(ctx)
go s.Run(ctx, lns)
defer func() { time.Sleep(3 * time.Second) }()
err = test.WaitForPortListen(fmt.Sprintf("127.0.0.1:%d", port), 10*time.Second)
assert.NoError(t, err)
// Allow server startup
time.Sleep(1 * time.Second)
client := http.Client{}
url := fmt.Sprintf("http://127.0.0.1:%d/test.html", port)
url := fmt.Sprintf("http://127.0.0.1:%d/test.html", s.ListenPort)
req, err := http.NewRequest("GET", url, nil)
assert.NoError(t, err)
resp, err := client.Do(req)
@@ -134,23 +127,19 @@ func Test_StaticHeaders(t *testing.T) {
s.ContentSecurityPolicy = "frame-ancestors 'none';"
cancelInformer := test.StartInformer(s.projInformer)
defer cancelInformer()
port, err := test.GetFreePort()
assert.NoError(t, err)
metricsPort, err := test.GetFreePort()
lns, err := s.Listen()
assert.NoError(t, err)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go s.Run(ctx, port, metricsPort)
s.Init(ctx)
go s.Run(ctx, lns)
defer func() { time.Sleep(3 * time.Second) }()
err = test.WaitForPortListen(fmt.Sprintf("127.0.0.1:%d", port), 10*time.Second)
assert.NoError(t, err)
// Allow server startup
time.Sleep(1 * time.Second)
client := http.Client{}
url := fmt.Sprintf("http://127.0.0.1:%d/test.html", port)
url := fmt.Sprintf("http://127.0.0.1:%d/test.html", s.ListenPort)
req, err := http.NewRequest("GET", url, nil)
assert.NoError(t, err)
resp, err := client.Do(req)
@@ -167,23 +156,22 @@ func Test_StaticHeaders(t *testing.T) {
s.ContentSecurityPolicy = ""
cancelInformer := test.StartInformer(s.projInformer)
defer cancelInformer()
port, err := test.GetFreePort()
assert.NoError(t, err)
metricsPort, err := test.GetFreePort()
lns, err := s.Listen()
assert.NoError(t, err)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go s.Run(ctx, port, metricsPort)
s.Init(ctx)
go s.Run(ctx, lns)
defer func() { time.Sleep(3 * time.Second) }()
err = test.WaitForPortListen(fmt.Sprintf("127.0.0.1:%d", port), 10*time.Second)
err = test.WaitForPortListen(fmt.Sprintf("127.0.0.1:%d", s.ListenPort), 10*time.Second)
assert.NoError(t, err)
// Allow server startup
time.Sleep(1 * time.Second)
client := http.Client{}
url := fmt.Sprintf("http://127.0.0.1:%d/test.html", port)
url := fmt.Sprintf("http://127.0.0.1:%d/test.html", s.ListenPort)
req, err := http.NewRequest("GET", url, nil)
assert.NoError(t, err)
resp, err := client.Do(req)

View File

@@ -41,8 +41,13 @@ func fakeServer() (*ArgoCDServer, func()) {
kubeclientset := fake.NewSimpleClientset(cm, secret)
appClientSet := apps.NewSimpleClientset()
redis, closer := test.NewInMemoryRedis()
port, err := test.GetFreePort()
if err != nil {
panic(err)
}
argoCDOpts := ArgoCDServerOpts{
ListenPort: port,
Namespace: test.FakeArgoCDNamespace,
KubeClientset: kubeclientset,
AppClientset: appClientSet,
@@ -61,7 +66,8 @@ func fakeServer() (*ArgoCDServer, func()) {
),
RedisClient: redis,
}
return NewServer(context.Background(), argoCDOpts), closer
srv := NewServer(context.Background(), argoCDOpts)
return srv, closer
}
func TestEnforceProjectToken(t *testing.T) {
@@ -508,7 +514,7 @@ func getTestServer(t *testing.T, anonymousEnabled bool, withFakeSSO bool) (argoc
cm.Data["users.anonymous.enabled"] = "true"
}
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
return // Start with a placeholder. We need the server URL before setting up the real handler.
// Start with a placeholder. We need the server URL before setting up the real handler.
}))
ts.Config.Handler = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
dexMockHandler(t, ts.URL)(w, r)
@@ -583,7 +589,7 @@ func TestAuthenticate_3rd_party_JWTs(t *testing.T) {
anonymousEnabled: false,
claims: jwt.RegisteredClaims{Audience: jwt.ClaimStrings{"test-client"}, Subject: "admin", ExpiresAt: jwt.NewNumericDate(time.Now())},
expectedErrorContains: "token is expired",
expectedClaims: jwt.RegisteredClaims{Issuer:"sso"},
expectedClaims: jwt.RegisteredClaims{Issuer: "sso"},
},
{
test: "anonymous enabled, expired token, admin claim",
@@ -601,7 +607,7 @@ func TestAuthenticate_3rd_party_JWTs(t *testing.T) {
t.Parallel()
// Must be declared here to avoid race.
ctx := context.Background() //nolint:ineffassign,staticcheck
ctx := context.Background() //nolint:ineffassign,staticcheck
argocd, dexURL := getTestServer(t, testDataCopy.anonymousEnabled, true)
testDataCopy.claims.Issuer = fmt.Sprintf("%s/api/dex", dexURL)
@@ -698,7 +704,7 @@ func TestAuthenticate_no_SSO(t *testing.T) {
t.Parallel()
// Must be declared here to avoid race.
ctx := context.Background() //nolint:ineffassign,staticcheck
ctx := context.Background() //nolint:ineffassign,staticcheck
argocd, dexURL := getTestServer(t, testDataCopy.anonymousEnabled, false)
token := jwt.NewWithClaims(jwt.SigningMethodHS256, jwt.RegisteredClaims{Issuer: fmt.Sprintf("%s/api/dex", dexURL)})
@@ -806,7 +812,7 @@ func TestAuthenticate_bad_request_metadata(t *testing.T) {
t.Parallel()
// Must be declared here to avoid race.
ctx := context.Background() //nolint:ineffassign,staticcheck
ctx := context.Background() //nolint:ineffassign,staticcheck
argocd, _ := getTestServer(t, testDataCopy.anonymousEnabled, true)
ctx = metadata.NewIncomingContext(context.Background(), testDataCopy.metadata)
@@ -1065,39 +1071,39 @@ func TestOIDCConfigChangeDetection_NoChange(t *testing.T) {
}
func TestIsMainJsBundle(t *testing.T) {
testCases := []struct{
testCases := []struct {
name string
url string
isMainJsBundle bool
}{
{
name: "localhost with valid main bundle",
url: "https://localhost:8080/main.e4188e5adc97bbfc00c3.js",
name: "localhost with valid main bundle",
url: "https://localhost:8080/main.e4188e5adc97bbfc00c3.js",
isMainJsBundle: true,
},
{
name: "localhost and deep path with valid main bundle",
url: "https://localhost:8080/some/argo-cd-instance/main.e4188e5adc97bbfc00c3.js",
name: "localhost and deep path with valid main bundle",
url: "https://localhost:8080/some/argo-cd-instance/main.e4188e5adc97bbfc00c3.js",
isMainJsBundle: true,
},
{
name: "font file",
url: "https://localhost:8080/assets/fonts/google-fonts/Heebo-Bols.woff2",
name: "font file",
url: "https://localhost:8080/assets/fonts/google-fonts/Heebo-Bols.woff2",
isMainJsBundle: false,
},
{
name: "no dot after main",
url: "https://localhost:8080/main/e4188e5adc97bbfc00c3.js",
name: "no dot after main",
url: "https://localhost:8080/main/e4188e5adc97bbfc00c3.js",
isMainJsBundle: false,
},
{
name: "wrong extension character",
url: "https://localhost:8080/main.e4188e5adc97bbfc00c3/js",
name: "wrong extension character",
url: "https://localhost:8080/main.e4188e5adc97bbfc00c3/js",
isMainJsBundle: false,
},
{
name: "wrong hash length",
url: "https://localhost:8080/main.e4188e5adc97bbfc00c3abcdefg.js",
name: "wrong hash length",
url: "https://localhost:8080/main.e4188e5adc97bbfc00c3abcdefg.js",
isMainJsBundle: false,
},
}
@@ -1111,3 +1117,123 @@ func TestIsMainJsBundle(t *testing.T) {
})
}
}
func TestReplaceBaseHRef(t *testing.T) {
testCases := []struct {
name string
data string
expected string
replaceWith string
}{
{
name: "non-root basepath",
data: `<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Argo CD</title>
<base href="/">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel='icon' type='image/png' href='assets/favicon/favicon-32x32.png' sizes='32x32'/>
<link rel='icon' type='image/png' href='assets/favicon/favicon-16x16.png' sizes='16x16'/>
<link href="assets/fonts.css" rel="stylesheet">
</head>
<body>
<noscript>
<p>
Your browser does not support JavaScript. Please enable JavaScript to view the site.
Alternatively, Argo CD can be used with the <a href="https://argoproj.github.io/argo-cd/cli_installation/">Argo CD CLI</a>.
</p>
</noscript>
<div id="app"></div>
</body>
</html>`,
expected: `<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Argo CD</title>
<base href="/path1/path2/path3/">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel='icon' type='image/png' href='assets/favicon/favicon-32x32.png' sizes='32x32'/>
<link rel='icon' type='image/png' href='assets/favicon/favicon-16x16.png' sizes='16x16'/>
<link href="assets/fonts.css" rel="stylesheet">
</head>
<body>
<noscript>
<p>
Your browser does not support JavaScript. Please enable JavaScript to view the site.
Alternatively, Argo CD can be used with the <a href="https://argoproj.github.io/argo-cd/cli_installation/">Argo CD CLI</a>.
</p>
</noscript>
<div id="app"></div>
</body>
</html>`,
replaceWith: `<base href="/path1/path2/path3/">`,
},
{
name: "root basepath",
data: `<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Argo CD</title>
<base href="/any/path/test/">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel='icon' type='image/png' href='assets/favicon/favicon-32x32.png' sizes='32x32'/>
<link rel='icon' type='image/png' href='assets/favicon/favicon-16x16.png' sizes='16x16'/>
<link href="assets/fonts.css" rel="stylesheet">
</head>
<body>
<noscript>
<p>
Your browser does not support JavaScript. Please enable JavaScript to view the site.
Alternatively, Argo CD can be used with the <a href="https://argoproj.github.io/argo-cd/cli_installation/">Argo CD CLI</a>.
</p>
</noscript>
<div id="app"></div>
</body>
</html>`,
expected: `<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Argo CD</title>
<base href="/">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel='icon' type='image/png' href='assets/favicon/favicon-32x32.png' sizes='32x32'/>
<link rel='icon' type='image/png' href='assets/favicon/favicon-16x16.png' sizes='16x16'/>
<link href="assets/fonts.css" rel="stylesheet">
</head>
<body>
<noscript>
<p>
Your browser does not support JavaScript. Please enable JavaScript to view the site.
Alternatively, Argo CD can be used with the <a href="https://argoproj.github.io/argo-cd/cli_installation/">Argo CD CLI</a>.
</p>
</noscript>
<div id="app"></div>
</body>
</html>`,
replaceWith: `<base href="/">`,
},
}
for _, testCase := range testCases {
t.Run(testCase.name, func(t *testing.T) {
result := replaceBaseHRef(testCase.data, testCase.replaceWith)
assert.Equal(t, testCase.expected, result)
})
}
}

View File

@@ -561,7 +561,9 @@ func EnsureCleanState(t *testing.T) {
FailOnErr(Run("", "mkdir", "-p", TmpDir))
// random id - unique across test runs
postFix := "-" + strings.ToLower(rand.RandString(5))
randString, err := rand.String(5)
CheckError(err)
postFix := "-" + strings.ToLower(randString)
id = t.Name() + postFix
name = DnsFriendly(t.Name(), "")
deploymentNamespace = DnsFriendly(fmt.Sprintf("argocd-e2e-%s", t.Name()), postFix)

View File

@@ -7,6 +7,7 @@ import (
"github.com/argoproj/gitops-engine/pkg/health"
. "github.com/argoproj/gitops-engine/pkg/sync/common"
"github.com/stretchr/testify/require"
. "github.com/argoproj/argo-cd/v2/pkg/apis/application/v1alpha1"
"github.com/argoproj/argo-cd/v2/test/e2e/fixture"
@@ -110,7 +111,9 @@ func TestSelectiveSyncWithNamespace(t *testing.T) {
}
func getNewNamespace(t *testing.T) string {
postFix := "-" + strings.ToLower(rand.RandString(5))
randStr, err := rand.String(5)
require.NoError(t, err)
postFix := "-" + strings.ToLower(randStr)
name := fixture.DnsFriendly(t.Name(), "")
return fixture.DnsFriendly(fmt.Sprintf("argocd-e2e-%s", name), postFix)
}

View File

@@ -1,9 +1,11 @@
ARG BASE_IMAGE=docker.io/library/ubuntu:22.04
FROM golang:1.18 AS go
RUN go install github.com/mattn/goreman@latest && \
go install github.com/kisielk/godepgraph@latest
FROM ubuntu:22.04
FROM $BASE_IMAGE
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install --no-install-recommends -y \

View File

@@ -0,0 +1,20 @@
import {ExternalLink, InvalidExternalLinkError} from './application-urls';
test('rejects malicious URLs', () => {
expect(() => {
const _ = new ExternalLink('javascript:alert("hi")');
}).toThrowError(InvalidExternalLinkError);
expect(() => {
const _ = new ExternalLink('data:text/html;<h1>hi</h1>');
}).toThrowError(InvalidExternalLinkError);
});
test('allows absolute URLs', () => {
expect(new ExternalLink('https://localhost:8080/applications').ref).toEqual('https://localhost:8080/applications');
});
test('allows relative URLs', () => {
// @ts-ignore
window.location = new URL('https://localhost:8080/applications');
expect(new ExternalLink('/applications').ref).toEqual('/applications');
});

View File

@@ -1,7 +1,15 @@
import {DropDownMenu} from 'argo-ui';
import * as React from 'react';
class ExternalLink {
export class InvalidExternalLinkError extends Error {
constructor(message: string) {
super(message);
Object.setPrototypeOf(this, InvalidExternalLinkError.prototype);
this.name = 'InvalidExternalLinkError';
}
}
export class ExternalLink {
public title: string;
public ref: string;
@@ -14,13 +22,36 @@ class ExternalLink {
this.title = url;
this.ref = url;
}
if (!ExternalLink.isValidURL(this.ref)) {
throw new InvalidExternalLinkError('Invalid URL');
}
}
private static isValidURL(url: string): boolean {
try {
const parsedUrl = new URL(url);
return parsedUrl.protocol !== 'javascript:' && parsedUrl.protocol !== 'data:';
} catch (TypeError) {
try {
// Try parsing as a relative URL.
const parsedUrl = new URL(url, window.location.origin);
return parsedUrl.protocol !== 'javascript:' && parsedUrl.protocol !== 'data:';
} catch (TypeError) {
return false;
}
}
}
}
export const ApplicationURLs = ({urls}: {urls: string[]}) => {
const externalLinks: ExternalLink[] = [];
for (const url of urls || []) {
externalLinks.push(new ExternalLink(url));
try {
const externalLink = new ExternalLink(url);
externalLinks.push(externalLink);
} catch (InvalidExternalLinkError) {
continue;
}
}
// sorted alphabetically & links with titles first

View File

@@ -141,8 +141,9 @@ export const PodTerminalViewer: React.FC<PodTerminalViewerProps> = ({selectedNod
function setupConnection() {
const {name = '', namespace = ''} = selectedNode || {};
const url = `${location.host}${appContext.baseHref}`.replace(/\/$/, '');
webSocket = new WebSocket(
`${location.protocol === 'https:' ? 'wss' : 'ws'}://${location.host}/terminal?pod=${name}&container=${AppUtils.getContainerName(
`${location.protocol === 'https:' ? 'wss' : 'ws'}://${url}/terminal?pod=${name}&container=${AppUtils.getContainerName(
podState,
activeContainer
)}&appName=${applicationName}&projectName=${projectName}&namespace=${namespace}`

View File

@@ -143,7 +143,7 @@ export class ApplicationsService {
const searchOptions = optionsToSearch(options);
search.set('fields', searchOptions.fields);
search.set('selector', searchOptions.selector);
query?.projects?.forEach(project => search.append('project', project));
query?.projects?.forEach(project => search.append('projects', project));
}
const searchStr = search.toString();
const url = `/stream/applications${(searchStr && '?' + searchStr) || ''}`;

View File

@@ -29,11 +29,11 @@ func IsManifestGenerationEnabled(sourceType v1alpha1.ApplicationSourceType, enab
return enabled
}
func Discover(ctx context.Context, repoPath string, enableGenerateManifests map[string]bool) (map[string]string, error) {
func Discover(ctx context.Context, repoPath string, enableGenerateManifests map[string]bool, tarExcludedGlobs []string) (map[string]string, error) {
apps := make(map[string]string)
// Check if it is CMP
conn, _, err := DetectConfigManagementPlugin(ctx, repoPath, []string{})
conn, _, err := DetectConfigManagementPlugin(ctx, repoPath, []string{}, tarExcludedGlobs)
if err == nil {
// Found CMP
io.Close(conn)
@@ -65,8 +65,8 @@ func Discover(ctx context.Context, repoPath string, enableGenerateManifests map[
return apps, err
}
func AppType(ctx context.Context, path string, enableGenerateManifests map[string]bool) (string, error) {
apps, err := Discover(ctx, path, enableGenerateManifests)
func AppType(ctx context.Context, path string, enableGenerateManifests map[string]bool, tarExcludedGlobs []string) (string, error) {
apps, err := Discover(ctx, path, enableGenerateManifests, tarExcludedGlobs)
if err != nil {
return "", err
}
@@ -82,7 +82,7 @@ func AppType(ctx context.Context, path string, enableGenerateManifests map[strin
// 3. check isSupported(path)?
// 4.a if no then close connection
// 4.b if yes then return conn for detected plugin
func DetectConfigManagementPlugin(ctx context.Context, repoPath string, env []string) (io.Closer, pluginclient.ConfigManagementPluginServiceClient, error) {
func DetectConfigManagementPlugin(ctx context.Context, repoPath string, env []string, tarExcludedGlobs []string) (io.Closer, pluginclient.ConfigManagementPluginServiceClient, error) {
var conn io.Closer
var cmpClient pluginclient.ConfigManagementPluginServiceClient
@@ -106,7 +106,7 @@ func DetectConfigManagementPlugin(ctx context.Context, repoPath string, env []st
continue
}
isSupported, err := matchRepositoryCMP(ctx, repoPath, cmpClient, env)
isSupported, err := matchRepositoryCMP(ctx, repoPath, cmpClient, env, tarExcludedGlobs)
if err != nil {
log.Errorf("repository %s is not the match because %v", repoPath, err)
continue
@@ -131,13 +131,13 @@ func DetectConfigManagementPlugin(ctx context.Context, repoPath string, env []st
// matchRepositoryCMP will send the repoPath to the cmp-server. The cmp-server will
// inspect the files and return true if the repo is supported for manifest generation.
// Will return false otherwise.
func matchRepositoryCMP(ctx context.Context, repoPath string, client pluginclient.ConfigManagementPluginServiceClient, env []string) (bool, error) {
func matchRepositoryCMP(ctx context.Context, repoPath string, client pluginclient.ConfigManagementPluginServiceClient, env []string, tarExcludedGlobs []string) (bool, error) {
matchRepoStream, err := client.MatchRepository(ctx, grpc_retry.Disable())
if err != nil {
return false, fmt.Errorf("error getting stream client: %s", err)
}
err = cmp.SendRepoStream(ctx, repoPath, repoPath, matchRepoStream, env)
err = cmp.SendRepoStream(ctx, repoPath, repoPath, matchRepoStream, env, tarExcludedGlobs)
if err != nil {
return false, fmt.Errorf("error sending stream: %s", err)
}

View File

@@ -10,7 +10,7 @@ import (
)
func TestDiscover(t *testing.T) {
apps, err := Discover(context.Background(), "./testdata", map[string]bool{})
apps, err := Discover(context.Background(), "./testdata", map[string]bool{}, []string{})
assert.NoError(t, err)
assert.Equal(t, map[string]string{
"foo": "Kustomize",
@@ -19,15 +19,15 @@ func TestDiscover(t *testing.T) {
}
func TestAppType(t *testing.T) {
appType, err := AppType(context.Background(), "./testdata/foo", map[string]bool{})
appType, err := AppType(context.Background(), "./testdata/foo", map[string]bool{}, []string{})
assert.NoError(t, err)
assert.Equal(t, "Kustomize", appType)
appType, err = AppType(context.Background(), "./testdata/baz", map[string]bool{})
appType, err = AppType(context.Background(), "./testdata/baz", map[string]bool{}, []string{})
assert.NoError(t, err)
assert.Equal(t, "Helm", appType)
appType, err = AppType(context.Background(), "./testdata", map[string]bool{})
appType, err = AppType(context.Background(), "./testdata", map[string]bool{}, []string{})
assert.NoError(t, err)
assert.Equal(t, "Directory", appType)
}
@@ -37,15 +37,15 @@ func TestAppType_Disabled(t *testing.T) {
string(v1alpha1.ApplicationSourceTypeKustomize): false,
string(v1alpha1.ApplicationSourceTypeHelm): false,
}
appType, err := AppType(context.Background(), "./testdata/foo", enableManifestGeneration)
appType, err := AppType(context.Background(), "./testdata/foo", enableManifestGeneration, []string{})
assert.NoError(t, err)
assert.Equal(t, "Directory", appType)
appType, err = AppType(context.Background(), "./testdata/baz", enableManifestGeneration)
appType, err = AppType(context.Background(), "./testdata/baz", enableManifestGeneration, []string{})
assert.NoError(t, err)
assert.Equal(t, "Directory", appType)
appType, err = AppType(context.Background(), "./testdata", enableManifestGeneration)
appType, err = AppType(context.Background(), "./testdata", enableManifestGeneration, []string{})
assert.NoError(t, err)
assert.Equal(t, "Directory", appType)
}

View File

@@ -84,11 +84,11 @@ func WithTarDoneChan(ch chan<- bool) SenderOption {
// SendRepoStream will compress the files under the given repoPath and send
// them using the plugin stream sender.
func SendRepoStream(ctx context.Context, appPath, repoPath string, sender StreamSender, env []string, opts ...SenderOption) error {
func SendRepoStream(ctx context.Context, appPath, repoPath string, sender StreamSender, env []string, excludedGlobs []string, opts ...SenderOption) error {
opt := newSenderOption(opts...)
// compress all files in appPath in tgz
tgz, checksum, err := compressFiles(repoPath)
tgz, checksum, err := compressFiles(repoPath, excludedGlobs)
if err != nil {
return fmt.Errorf("error compressing repo files: %w", err)
}
@@ -162,11 +162,10 @@ func closeAndDelete(f *os.File) {
}
// compressFiles will create a tgz file with all contents of appPath
// directory excluding the .git folder. Returns the file alongside
// its sha256 hash to be used as checksum. It is the responsibility
// of the caller to close the file.
func compressFiles(appPath string) (*os.File, string, error) {
excluded := []string{".git"}
// directory excluding globs in the excluded array. Returns the file
// alongside its sha256 hash to be used as checksum. It is the
// responsibility of the caller to close the file.
func compressFiles(appPath string, excluded []string) (*os.File, string, error) {
appName := filepath.Base(appPath)
tempDir, err := files.CreateTempDir(os.TempDir())
if err != nil {

View File

@@ -60,7 +60,7 @@ func TestReceiveApplicationStream(t *testing.T) {
close(streamMock.messages)
os.RemoveAll(workdir)
}()
go streamMock.sendFile(context.Background(), t, appDir, streamMock, []string{"env1", "env2"})
go streamMock.sendFile(context.Background(), t, appDir, streamMock, []string{"env1", "env2"}, []string{"DUMMY.md", "dum*"})
// when
env, err := cmp.ReceiveRepoStream(context.Background(), streamMock, workdir)
@@ -77,16 +77,18 @@ func TestReceiveApplicationStream(t *testing.T) {
}
assert.Contains(t, names, "README.md")
assert.Contains(t, names, "applicationset")
assert.NotContains(t, names, "DUMMY.md")
assert.NotContains(t, names, "dummy")
assert.NotNil(t, env)
})
}
func (m *streamMock) sendFile(ctx context.Context, t *testing.T, basedir string, sender cmp.StreamSender, env []string) {
func (m *streamMock) sendFile(ctx context.Context, t *testing.T, basedir string, sender cmp.StreamSender, env []string, excludedGlobs []string) {
t.Helper()
defer func() {
m.done <- true
}()
err := cmp.SendRepoStream(ctx, basedir, basedir, sender, env)
err := cmp.SendRepoStream(ctx, basedir, basedir, sender, env, excludedGlobs)
require.NoError(t, err)
}

1
util/cmp/testdata/app/DUMMY.md vendored Normal file
View File

@@ -0,0 +1 @@
This file is used to test the tar stream exclusions.

1
util/cmp/testdata/app/dummy/DUMMY2.md vendored Normal file
View File

@@ -0,0 +1 @@
This is another file that should get excluded because the entire directory is excluded.

7
util/env/env.go vendored
View File

@@ -126,6 +126,13 @@ func StringFromEnv(env string, defaultValue string) string {
return defaultValue
}
func StringsFromEnv(env string, defaultValue []string, separator string) []string {
if str := os.Getenv(env); str != "" {
return strings.Split(str, separator)
}
return defaultValue
}
// ParseBoolFromEnv retrieves a boolean value from given environment envVar.
// Returns default value if envVar is not set.
//

View File

@@ -326,6 +326,16 @@ func (m *nativeGitClient) IsLFSEnabled() bool {
return m.enableLfs
}
func (m *nativeGitClient) fetch(revision string) error {
var err error
if revision != "" {
err = m.runCredentialedCmd("git", "fetch", "origin", revision, "--tags", "--force")
} else {
err = m.runCredentialedCmd("git", "fetch", "origin", "--tags", "--force")
}
return err
}
// Fetch fetches latest updates from origin
func (m *nativeGitClient) Fetch(revision string) error {
if m.OnFetch != nil {
@@ -334,11 +344,19 @@ func (m *nativeGitClient) Fetch(revision string) error {
}
var err error
if revision != "" {
err = m.runCredentialedCmd("git", "fetch", "origin", revision, "--tags", "--force")
} else {
err = m.runCredentialedCmd("git", "fetch", "origin", "--tags", "--force")
err = m.fetch(revision)
if err != nil {
errMsg := strings.ReplaceAll(err.Error(), "\n", "")
if strings.Contains(errMsg, "try running 'git remote prune origin'") {
// Prune any deleted refs, then try fetching again
if err := m.runCredentialedCmd("git", "remote", "prune", "origin"); err != nil {
return err
}
err = m.fetch(revision)
}
}
// When we have LFS support enabled, check for large files and fetch them too.
if err == nil && m.IsLFSEnabled() {
largeFiles, err := m.LsLargeFiles()
@@ -349,6 +367,7 @@ func (m *nativeGitClient) Fetch(revision string) error {
}
}
}
return err
}

94
util/git/client_test.go Normal file
View File

@@ -0,0 +1,94 @@
package git
import (
"fmt"
"os"
"os/exec"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func Test_nativeGitClient_Fetch(t *testing.T) {
tempDir, err := os.MkdirTemp("", "")
require.NoError(t, err)
cmd := exec.Command("git", "init")
cmd.Dir = tempDir
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err = cmd.Run()
require.NoError(t, err)
cmd = exec.Command("git", "commit", "-m", "Initial commit", "--allow-empty")
cmd.Dir = tempDir
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err = cmd.Run()
require.NoError(t, err)
client, err := NewClient(fmt.Sprintf("file://%s", tempDir), NopCreds{}, true, false, "")
require.NoError(t, err)
err = client.Init()
require.NoError(t, err)
err = client.Fetch("")
assert.NoError(t, err)
}
func Test_nativeGitClient_Fetch_Prune(t *testing.T) {
tempDir, err := os.MkdirTemp("", "")
require.NoError(t, err)
cmd := exec.Command("git", "init")
cmd.Dir = tempDir
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err = cmd.Run()
require.NoError(t, err)
cmd = exec.Command("git", "commit", "-m", "Initial commit", "--allow-empty")
cmd.Dir = tempDir
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err = cmd.Run()
require.NoError(t, err)
client, err := NewClient(fmt.Sprintf("file://%s", tempDir), NopCreds{}, true, false, "")
require.NoError(t, err)
err = client.Init()
require.NoError(t, err)
// Create branch
cmd = exec.Command("git", "branch", "test/foo")
cmd.Dir = tempDir
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err = cmd.Run()
require.NoError(t, err)
err = client.Fetch("")
assert.NoError(t, err)
// Delete branch
cmd = exec.Command("git", "branch", "-d", "test/foo")
cmd.Dir = tempDir
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err = cmd.Run()
require.NoError(t, err)
// Create branch that conflicts with previous branch name
cmd = exec.Command("git", "branch", "test/foo/bar")
cmd.Dir = tempDir
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err = cmd.Run()
require.NoError(t, err)
err = client.Fetch("")
assert.NoError(t, err)
}

View File

@@ -6,9 +6,11 @@ import (
"net/url"
"os"
"os/exec"
"path/filepath"
"strings"
"github.com/ghodss/yaml"
log "github.com/sirupsen/logrus"
"github.com/argoproj/argo-cd/v2/util/config"
executil "github.com/argoproj/argo-cd/v2/util/exec"
@@ -27,7 +29,7 @@ type Helm interface {
// Template returns a list of unstructured objects from a `helm template` command
Template(opts *TemplateOpts) (string, error)
// GetParameters returns a list of chart parameters taking into account values in provided YAML files.
GetParameters(valuesFiles []pathutil.ResolvedFilePath) (map[string]string, error)
GetParameters(valuesFiles []pathutil.ResolvedFilePath, appPath, repoRoot string) (map[string]string, error)
// DependencyBuild runs `helm dependency build` to download a chart's dependencies
DependencyBuild() error
// Init runs `helm init --client-only`
@@ -129,12 +131,19 @@ func Version(shortForm bool) (string, error) {
return strings.TrimSpace(version), nil
}
func (h *helm) GetParameters(valuesFiles []pathutil.ResolvedFilePath) (map[string]string, error) {
out, err := h.cmd.inspectValues(".")
if err != nil {
return nil, err
func (h *helm) GetParameters(valuesFiles []pathutil.ResolvedFilePath, appPath, repoRoot string) (map[string]string, error) {
var values []string
// Don't load values.yaml if it's an out-of-bounds link.
if resolved, _, err := pathutil.ResolveFilePath(appPath, repoRoot, "values.yaml", []string{}); err == nil {
fmt.Println(resolved)
out, err := h.cmd.inspectValues(".")
if err != nil {
return nil, err
}
values = append(values, out)
} else {
log.Warnf("Values file %s is not allowed: %v", filepath.Join(appPath, "values.yaml"), err)
}
values := []string{out}
for i := range valuesFiles {
file := string(valuesFiles[i])
var fileValues []byte
@@ -156,7 +165,7 @@ func (h *helm) GetParameters(valuesFiles []pathutil.ResolvedFilePath) (map[strin
output := map[string]string{}
for _, file := range values {
values := map[string]interface{}{}
if err = yaml.Unmarshal([]byte(file), &values); err != nil {
if err := yaml.Unmarshal([]byte(file), &values); err != nil {
return nil, fmt.Errorf("failed to parse values: %s", err)
}
flatVals(values, output)

View File

@@ -85,7 +85,7 @@ func TestHelmGetParams(t *testing.T) {
require.NoError(t, err)
h, err := NewHelmApp(repoRootAbs, nil, false, "", "", false)
assert.NoError(t, err)
params, err := h.GetParameters(nil)
params, err := h.GetParameters(nil, repoRootAbs, repoRootAbs)
assert.Nil(t, err)
slaveCountParam := params["cluster.slaveCount"]
@@ -100,7 +100,7 @@ func TestHelmGetParamsValueFiles(t *testing.T) {
assert.NoError(t, err)
valuesPath, _, err := path.ResolveFilePath(repoRootAbs, repoRootAbs, "values-production.yaml", nil)
require.NoError(t, err)
params, err := h.GetParameters([]path.ResolvedFilePath{valuesPath})
params, err := h.GetParameters([]path.ResolvedFilePath{valuesPath}, repoRootAbs, repoRootAbs)
assert.Nil(t, err)
slaveCountParam := params["cluster.slaveCount"]
@@ -117,7 +117,7 @@ func TestHelmGetParamsValueFilesThatExist(t *testing.T) {
require.NoError(t, err)
valuesProductionPath, _, err := path.ResolveFilePath(repoRootAbs, repoRootAbs, "values-production.yaml", nil)
require.NoError(t, err)
params, err := h.GetParameters([]path.ResolvedFilePath{valuesMissingPath, valuesProductionPath})
params, err := h.GetParameters([]path.ResolvedFilePath{valuesMissingPath, valuesProductionPath}, repoRootAbs, repoRootAbs)
assert.Nil(t, err)
slaveCountParam := params["cluster.slaveCount"]

Some files were not shown because too many files have changed in this diff Show More