{
"file_path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/ISSUE.md"
}1 **Title: kubectl exec interactive sessions fail due to missing session uploader initialization in Kubernetes service **
2
3 **Expected behavior:**
4 When using the Kubernetes integration in Teleport, executing `kubectl exec` against a pod should open an interactive shell without requiring manual setup of log directories or additional configuration steps.
5
6 **Current behavior:**
7 Attempts to run `kubectl exec` do not open a shell. Instead, the execution fails with a warning in the logs indicating that the path `/var/lib/teleport/log/upload/streaming/default` does not exist or is not a directory. This causes all session recordings to fail and prevents interactive sessions from being established.
8
9 **Bug details:**
10 The Kubernetes service was missing initialization of the session uploader, which is required to create the async upload directory on disk. As a result, sessions relying on that path fail due to the missing directory.
11
12 The `clusterSession` object was being fully cached, including request-specific and cluster-related state that should not persist. This introduced complications, especially when remote clusters or tunnels disappear.
13
14 Audit events were emitted using the request context, which can be prematurely canceled when the client disconnects, leading to missing audit events. Logging of response errors from the exec handler was incomplete. Config fields in the Kubernetes forwarder were inconsistently named or embedded unnecessarily, making the API harder to maintain.
15
16 **Recreation steps:**
17 1. Deploy `teleport-kube-agent` using the provided example Helm chart.
18 2. Execute `kubectl exec` on a running pod.
19 3. Observe that no shell is opened.
20 4. Check Teleport server logs and find errors indicating the session log path is missing.
21 5. Workaround involves manually creating the missing directory:
22
23 `mkdir -p /var/lib/teleport/log/upload/streaming/default`
24
25 **Debug logs:**
26 ```
27 WARN [PROXY:PRO] Executor failed while streaming. error:path "/var/lib/teleport/log/upload/streaming/default" does not exist or is not a directory proxy/forwarder.go:773
28 ```{
"pattern": "**/forwarder.go",
"path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037"
}lib/kube/proxy/forwarder.go{
"pattern": "upload|StreamingSessionsDir|sessionUploader|NewStreamer",
"path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/kube/proxy/forwarder.go",
"output_mode": "content",
"context": 3
}565-// newStreamer returns sync or async streamer based on the configuration
566-// of the server and the session, sync streamer sends the events
567-// directly to the auth server and blocks if the events can not be received,
568:// async streamer buffers the events to disk and uploads the events later
569-func (f *Forwarder) newStreamer(ctx *authContext) (events.Streamer, error) {
570- mode := ctx.clusterConfig.GetSessionRecording()
571- if services.IsRecordSync(mode) {
--
577- f.DataDir, teleport.LogsDir, teleport.ComponentUpload,
578- events.StreamingLogsDir, defaults.Namespace,
579- )
580: fileStreamer, err := filesessions.NewStreamer(dir)
581- if err != nil {
582- return nil, trace.Wrap(err)
583- }
584- // TeeStreamer sends non-print and non disk events
585- // to the audit log in async mode, while buffering all
586: // events on disk for further upload at the end of the session
587- return events.NewTeeStreamer(fileStreamer, f.StreamEmitter), nil
588-}
589-
--
636- // node so we don't create double recordings.
637- recorder, err = events.NewAuditWriter(events.AuditWriterConfig{
638- // Audit stream is using server context, not session context,
639: // to make sure that session is uploaded even after it is closed
640- Context: request.context,
641- Streamer: streamer,
642- Clock: f.Clock,{
"pattern": "NewUploader\\|newUploader\\|sessionUploader\\|InitUploader\\|initUploader\\|UploaderConfig\\|uploaderConfig",
"path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/kube/proxy",
"output_mode": "content",
"context": 2
}No matches found{
"pattern": "NewUploader\\|sessionUploader\\|filesessions.NewUploader",
"path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037",
"output_mode": "content",
"context": 2
}No matches found{
"pattern": "StreamingLogsDir\\|StreamingSessionsDir\\|upload/streaming",
"path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037",
"output_mode": "content",
"context": 3
}No matches found{
"pattern": "StreamingLogsDir",
"path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037",
"output_mode": "content",
"context": 2
}/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/srv/sess.go-1069- return filepath.Join(
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/srv/sess.go-1070- ctx.srv.GetDataDir(), teleport.LogsDir, teleport.ComponentUpload,
lib/srv/sess.go:1071: events.StreamingLogsDir, ctx.srv.GetNamespace(),
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/srv/sess.go-1072- )
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/srv/sess.go-1073-}
--
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/srv/app/server_test.go-178- err = os.MkdirAll(filepath.Join(
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/srv/app/server_test.go-179- s.dataDir, teleport.LogsDir, teleport.ComponentUpload,
lib/srv/app/server_test.go:180: events.StreamingLogsDir, defaults.Namespace,
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/srv/app/server_test.go-181- ), 0755)
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/srv/app/server_test.go-182- c.Assert(err, check.IsNil)
--
lib/srv/app/session.go-169- uploadDir := filepath.Join(
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/srv/app/session.go-170- s.c.DataDir, teleport.LogsDir, teleport.ComponentUpload,
lib/srv/app/session.go:171: events.StreamingLogsDir, defaults.Namespace,
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/srv/app/session.go-172- )
lib/srv/app/session.go-173- fileStreamer, err := filesessions.NewStreamer(uploadDir)
--
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/auditlog.go-49- SessionLogsDir = "sessions"
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/auditlog.go-50-
lib/events/auditlog.go:51: // StreamingLogsDir is a subdirectory of sessions /var/lib/teleport/logs/streaming
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/auditlog.go-52- // is used in new versions of the uploader
lib/events/auditlog.go:53: StreamingLogsDir = "streaming"
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/auditlog.go-54-
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/auditlog.go-55- // RecordsDir is a subdirectory with default records /var/lib/teleport/logs/records
--
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/service/service.go-1850- }
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/service/service.go-1851- // prepare dirs for uploader
lib/service/service.go:1852: streamingDir := []string{process.Config.DataDir, teleport.LogsDir, teleport.ComponentUpload, events.StreamingLogsDir, defaults.Namespace}
lib/service/service.go-1853- paths := [][]string{
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/service/service.go-1854- // DELETE IN (5.1.0)
--
lib/kube/proxy/forwarder.go-576- dir := filepath.Join(
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/kube/proxy/forwarder.go-577- f.DataDir, teleport.LogsDir, teleport.ComponentUpload,
lib/kube/proxy/forwarder.go:578: events.StreamingLogsDir, defaults.Namespace,
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/kube/proxy/forwarder.go-579- )
lib/kube/proxy/forwarder.go-580- fileStreamer, err := filesessions.NewStreamer(dir)
--
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/auth/helpers.go-86-
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/auth/helpers.go-87- err = os.MkdirAll(filepath.Join(dir, teleport.LogsDir, teleport.ComponentUpload,
lib/auth/helpers.go:88: events.StreamingLogsDir, defaults.Namespace), teleport.SharedDirMode)
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/auth/helpers.go-89- if err != nil {
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/auth/helpers.go-90- return trace.ConvertSystemError(err){
"file_path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/kube/proxy/forwarder.go",
"offset": 560,
"limit": 50
}560 return nil
561 }
562 return trace.AccessDenied("kubernetes cluster %q not found", actx.kubeCluster)
563 }
564
565 // newStreamer returns sync or async streamer based on the configuration
566 // of the server and the session, sync streamer sends the events
567 // directly to the auth server and blocks if the events can not be received,
568 // async streamer buffers the events to disk and uploads the events later
569 func (f *Forwarder) newStreamer(ctx *authContext) (events.Streamer, error) {
570 mode := ctx.clusterConfig.GetSessionRecording()
571 if services.IsRecordSync(mode) {
572 f.log.Debugf("Using sync streamer for session.")
573 return f.Client, nil
574 }
575 f.log.Debugf("Using async streamer for session.")
576 dir := filepath.Join(
577 f.DataDir, teleport.LogsDir, teleport.ComponentUpload,
578 events.StreamingLogsDir, defaults.Namespace,
579 )
580 fileStreamer, err := filesessions.NewStreamer(dir)
581 if err != nil {
582 return nil, trace.Wrap(err)
583 }
584 // TeeStreamer sends non-print and non disk events
585 // to the audit log in async mode, while buffering all
586 // events on disk for further upload at the end of the session
587 return events.NewTeeStreamer(fileStreamer, f.StreamEmitter), nil
588 }
589
590 // exec forwards all exec requests to the target server, captures
591 // all output from the session
592 func (f *Forwarder) exec(ctx *authContext, w http.ResponseWriter, req *http.Request, p httprouter.Params) (interface{}, error) {
593 f.log.Debugf("Exec %v.", req.URL.String())
594
595 sess, err := f.getOrCreateClusterSession(*ctx)
596 if err != nil {
597 // This error goes to kubernetes client and is not visible in the logs
598 // of the teleport server if not logged here.
599 f.log.Errorf("Failed to create cluster session: %v.", err)
600 return nil, trace.Wrap(err)
601 }
602 sessionStart := f.Clock.Now().UTC()
603
604 q := req.URL.Query()
605 request := remoteCommandRequest{
606 podNamespace: p.ByName("podNamespace"),
607 podName: p.ByName("podName"),
608 containerName: q.Get("container"),
609 cmd: q["command"],{
"file_path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/srv/app/session.go",
"offset": 160,
"limit": 30
}160 // async streamer buffers the events to disk and uploads the events later
161 func (s *Server) newStreamer(ctx context.Context, sessionID string, clusterConfig services.ClusterConfig) (events.Streamer, error) {
162 mode := clusterConfig.GetSessionRecording()
163 if services.IsRecordSync(mode) {
164 s.log.Debugf("Using sync streamer for session %v.", sessionID)
165 return s.c.AuthClient, nil
166 }
167
168 s.log.Debugf("Using async streamer for session %v.", sessionID)
169 uploadDir := filepath.Join(
170 s.c.DataDir, teleport.LogsDir, teleport.ComponentUpload,
171 events.StreamingLogsDir, defaults.Namespace,
172 )
173 fileStreamer, err := filesessions.NewStreamer(uploadDir)
174 if err != nil {
175 return nil, trace.Wrap(err)
176 }
177 return fileStreamer, nil
178 }
179
180 // sessionCache holds a cache of sessions that are used to forward requests.
181 type sessionCache struct {
182 mu sync.Mutex
183 cache *ttlmap.TTLMap
184
185 closeContext context.Context
186
187 log *logrus.Entry
188 }
189 {
"pattern": "initKubernetes\\|KubernetesService\\|newKubeForwarder\\|NewForwarder\\|kube.*service\\|kubeService",
"path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/service/service.go",
"output_mode": "content",
"context": 2
}No matches found{
"pattern": "kube\\|Kubernetes",
"path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/service/service.go",
"output_mode": "content",
"context": 1,
"head_limit": 50
}No matches found{
"file_path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/service/service.go",
"offset": 1840,
"limit": 60
}1840 }
1841
1842 func (process *TeleportProcess) initUploaderService(accessPoint auth.AccessPoint, auditLog events.IAuditLog) error {
1843 log := process.log.WithFields(logrus.Fields{
1844 trace.Component: teleport.Component(teleport.ComponentAuditLog, process.id),
1845 })
1846 // create folder for uploads
1847 uid, gid, err := adminCreds()
1848 if err != nil {
1849 return trace.Wrap(err)
1850 }
1851 // prepare dirs for uploader
1852 streamingDir := []string{process.Config.DataDir, teleport.LogsDir, teleport.ComponentUpload, events.StreamingLogsDir, defaults.Namespace}
1853 paths := [][]string{
1854 // DELETE IN (5.1.0)
1855 // this directory will no longer be used after migration to 5.1.0
1856 []string{process.Config.DataDir, teleport.LogsDir, teleport.ComponentUpload, events.SessionLogsDir, defaults.Namespace},
1857 // This directory will remain to be used after migration to 5.1.0
1858 streamingDir,
1859 }
1860 for _, path := range paths {
1861 for i := 1; i < len(path); i++ {
1862 dir := filepath.Join(path[:i+1]...)
1863 log.Infof("Creating directory %v.", dir)
1864 err := os.Mkdir(dir, 0755)
1865 err = trace.ConvertSystemError(err)
1866 if err != nil {
1867 if !trace.IsAlreadyExists(err) {
1868 return trace.Wrap(err)
1869 }
1870 }
1871 if uid != nil && gid != nil {
1872 log.Infof("Setting directory %v owner to %v:%v.", dir, *uid, *gid)
1873 err := os.Chown(dir, *uid, *gid)
1874 if err != nil {
1875 return trace.ConvertSystemError(err)
1876 }
1877 }
1878 }
1879 }
1880
1881 // DELETE IN (5.1.0)
1882 // this uploader was superseded by filesessions.Uploader,
1883 // see below
1884 uploader, err := events.NewUploader(events.UploaderConfig{
1885 DataDir: filepath.Join(process.Config.DataDir, teleport.LogsDir),
1886 Namespace: defaults.Namespace,
1887 ServerID: teleport.ComponentUpload,
1888 AuditLog: auditLog,
1889 EventsC: process.Config.UploadEventsC,
1890 })
1891 if err != nil {
1892 return trace.Wrap(err)
1893 }
1894 process.RegisterFunc("uploader.service", func() error {
1895 err := uploader.Serve()
1896 if err != nil {
1897 log.Errorf("Uploader server exited with error: %v.", err)
1898 }
1899 return nil{
"file_path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/service/service.go",
"offset": 1900,
"limit": 100
}1900 })
1901
1902 process.onExit("uploader.shutdown", func(payload interface{}) {
1903 log.Infof("Shutting down.")
1904 warnOnErr(uploader.Stop(), log)
1905 log.Infof("Exited.")
1906 })
1907
1908 // This uploader supersedes the events.Uploader above,
1909 // that is kept for backwards compatibility purposes for one release.
1910 // Delete this comment once the uploader above is phased out.
1911 fileUploader, err := filesessions.NewUploader(filesessions.UploaderConfig{
1912 ScanDir: filepath.Join(streamingDir...),
1913 Streamer: accessPoint,
1914 EventsC: process.Config.UploadEventsC,
1915 })
1916 if err != nil {
1917 return trace.Wrap(err)
1918 }
1919 process.RegisterFunc("fileuploader.service", func() error {
1920 err := fileUploader.Serve()
1921 if err != nil {
1922 log.WithError(err).Errorf("File uploader server exited with error.")
1923 }
1924 return nil
1925 })
1926
1927 process.onExit("fileuploader.shutdown", func(payload interface{}) {
1928 log.Infof("File uploader is shutting down.")
1929 warnOnErr(fileUploader.Close(), log)
1930 log.Infof("File uploader has shut down.")
1931 })
1932
1933 return nil
1934 }
1935
1936 // initDiagnosticService starts diagnostic service currently serving healthz
1937 // and prometheus endpoints
1938 func (process *TeleportProcess) initDiagnosticService() error {
1939 mux := http.NewServeMux()
1940 mux.Handle("/metrics", promhttp.Handler())
1941
1942 if process.Config.Debug {
1943 process.log.Infof("Adding diagnostic debugging handlers. To connect with profiler, use `go tool pprof %v`.", process.Config.DiagnosticAddr.Addr)
1944
1945 mux.HandleFunc("/debug/pprof/", pprof.Index)
1946 mux.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
1947 mux.HandleFunc("/debug/pprof/profile", pprof.Profile)
1948 mux.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
1949 mux.HandleFunc("/debug/pprof/trace", pprof.Trace)
1950 }
1951
1952 mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
1953 roundtrip.ReplyJSON(w, http.StatusOK, map[string]interface{}{"status": "ok"})
1954 })
1955
1956 log := process.log.WithFields(logrus.Fields{
1957 trace.Component: teleport.Component(teleport.ComponentDiagnostic, process.id),
1958 })
1959
1960 // Create a state machine that will process and update the internal state of
1961 // Teleport based off Events. Use this state machine to return return the
1962 // status from the /readyz endpoint.
1963 ps := newProcessState(process)
1964 process.RegisterFunc("readyz.monitor", func() error {
1965 // Start loop to monitor for events that are used to update Teleport state.
1966 eventCh := make(chan Event, 1024)
1967 process.WaitForEvent(process.ExitContext(), TeleportDegradedEvent, eventCh)
1968 process.WaitForEvent(process.ExitContext(), TeleportOKEvent, eventCh)
1969
1970 for {
1971 select {
1972 case e := <-eventCh:
1973 ps.update(e)
1974 case <-process.ExitContext().Done():
1975 log.Debugf("Teleport is exiting, returning.")
1976 return nil
1977 }
1978 }
1979 })
1980 mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
1981 switch ps.getState() {
1982 // 503
1983 case stateDegraded:
1984 roundtrip.ReplyJSON(w, http.StatusServiceUnavailable, map[string]interface{}{
1985 "status": "teleport is in a degraded state, check logs for details",
1986 })
1987 // 400
1988 case stateRecovering:
1989 roundtrip.ReplyJSON(w, http.StatusBadRequest, map[string]interface{}{
1990 "status": "teleport is recovering from a degraded state, check logs for details",
1991 })
1992 case stateStarting:
1993 roundtrip.ReplyJSON(w, http.StatusBadRequest, map[string]interface{}{
1994 "status": "teleport is starting and hasn't joined the cluster yet",
1995 })
1996 // 200
1997 case stateOK:
1998 roundtrip.ReplyJSON(w, http.StatusOK, map[string]interface{}{
1999 "status": "ok",{
"pattern": "func NewForwarder\\|func.*NewForwarder",
"path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/kube/proxy/forwarder.go",
"output_mode": "content",
"context": 3
}No matches found{
"pattern": "func New",
"path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/kube/proxy/forwarder.go",
"output_mode": "content",
"context": 3
}165-
166-// NewForwarder returns new instance of Kubernetes request
167-// forwarding proxy.
168:func NewForwarder(cfg ForwarderConfig) (*Forwarder, error) {
169- if err := cfg.CheckAndSetDefaults(); err != nil {
170- return nil, trace.Wrap(err)
171- }{
"file_path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/kube/proxy/forwarder.go",
"offset": 166,
"limit": 80
}166 // NewForwarder returns new instance of Kubernetes request
167 // forwarding proxy.
168 func NewForwarder(cfg ForwarderConfig) (*Forwarder, error) {
169 if err := cfg.CheckAndSetDefaults(); err != nil {
170 return nil, trace.Wrap(err)
171 }
172 log := log.WithFields(log.Fields{
173 trace.Component: cfg.Component,
174 })
175
176 creds, err := getKubeCreds(cfg.Context, log, cfg.ClusterName, cfg.KubeClusterName, cfg.KubeconfigPath, cfg.NewKubeService)
177 if err != nil {
178 return nil, trace.Wrap(err)
179 }
180
181 clusterSessions, err := ttlmap.New(defaults.ClientCacheSize)
182 if err != nil {
183 return nil, trace.Wrap(err)
184 }
185 closeCtx, close := context.WithCancel(cfg.Context)
186 fwd := &Forwarder{
187 creds: creds,
188 log: log,
189 Router: *httprouter.New(),
190 ForwarderConfig: cfg,
191 clusterSessions: clusterSessions,
192 activeRequests: make(map[string]context.Context),
193 ctx: closeCtx,
194 close: close,
195 }
196
197 fwd.POST("/api/:ver/namespaces/:podNamespace/pods/:podName/exec", fwd.withAuth(fwd.exec))
198 fwd.GET("/api/:ver/namespaces/:podNamespace/pods/:podName/exec", fwd.withAuth(fwd.exec))
199
200 fwd.POST("/api/:ver/namespaces/:podNamespace/pods/:podName/attach", fwd.withAuth(fwd.exec))
201 fwd.GET("/api/:ver/namespaces/:podNamespace/pods/:podName/attach", fwd.withAuth(fwd.exec))
202
203 fwd.POST("/api/:ver/namespaces/:podNamespace/pods/:podName/portforward", fwd.withAuth(fwd.portForward))
204 fwd.GET("/api/:ver/namespaces/:podNamespace/pods/:podName/portforward", fwd.withAuth(fwd.portForward))
205
206 fwd.NotFound = fwd.withAuthStd(fwd.catchAll)
207
208 if cfg.ClusterOverride != "" {
209 fwd.log.Debugf("Cluster override is set, forwarder will send all requests to remote cluster %v.", cfg.ClusterOverride)
210 }
211 return fwd, nil
212 }
213
214 // Forwarder intercepts kubernetes requests, acting as Kubernetes API proxy.
215 // it blindly forwards most of the requests on HTTPS protocol layer,
216 // however some requests like exec sessions it intercepts and records.
217 type Forwarder struct {
218 sync.Mutex
219 httprouter.Router
220 ForwarderConfig
221
222 // log specifies the logger
223 log log.FieldLogger
224 // clusterSessions is an expiring cache associated with authenticated
225 // user connected to a remote cluster, session is invalidated
226 // if user changes kubernetes groups via RBAC or cache has expired
227 // TODO(klizhentas): flush certs on teleport CA rotation?
228 clusterSessions *ttlmap.TTLMap
229 // activeRequests is a map used to serialize active CSR requests to the auth server
230 activeRequests map[string]context.Context
231 // close is a close function
232 close context.CancelFunc
233 // ctx is a global context signalling exit
234 ctx context.Context
235 // creds contain kubernetes credentials for multiple clusters.
236 // map key is cluster name.
237 creds map[string]*kubeCreds
238 }
239
240 // Close signals close to all outstanding or background operations
241 // to complete
242 func (f *Forwarder) Close() error {
243 f.close()
244 return nil
245 }{
"pattern": "func NewStreamer",
"path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037",
"output_mode": "content",
"context": 5
}/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/filestream.go-35- "github.com/gravitational/trace"
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/filestream.go-36- "github.com/pborman/uuid"
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/filestream.go-37-)
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/filestream.go-38-
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/filestream.go-39-// NewStreamer creates a streamer sending uploads to disk
lib/events/filesessions/filestream.go:40:func NewStreamer(dir string) (*events.ProtoStreamer, error) {
lib/events/filesessions/filestream.go-41- handler, err := NewHandler(Config{
lib/events/filesessions/filestream.go-42- Directory: dir,
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/filestream.go-43- })
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/filestream.go-44- if err != nil {
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/filestream.go-45- return nil, trace.Wrap(err){
"pattern": "func NewHandler",
"path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions",
"output_mode": "content",
"context": 10
}/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/fileuploader.go-54- if utils.IsDir(s.Directory) == false {
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/fileuploader.go-55- return trace.BadParameter("path %q does not exist or is not a directory", s.Directory)
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/fileuploader.go-56- }
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/fileuploader.go-57- if s.OnBeforeComplete == nil {
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/fileuploader.go-58- s.OnBeforeComplete = nopBeforeComplete
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/fileuploader.go-59- }
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/fileuploader.go-60- return nil
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/fileuploader.go-61-}
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/fileuploader.go-62-
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/fileuploader.go-63-// NewHandler returns new file sessions handler
lib/events/filesessions/fileuploader.go:64:func NewHandler(cfg Config) (*Handler, error) {
lib/events/filesessions/fileuploader.go-65- if err := cfg.CheckAndSetDefaults(); err != nil {
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/fileuploader.go-66- return nil, trace.Wrap(err)
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/fileuploader.go-67- }
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/fileuploader.go-68-
lib/events/filesessions/fileuploader.go-69- h := &Handler{
lib/events/filesessions/fileuploader.go-70- Entry: log.WithFields(log.Fields{
lib/events/filesessions/fileuploader.go-71- trace.Component: teleport.Component(teleport.SchemeFile),
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/fileuploader.go-72- }),
lib/events/filesessions/fileuploader.go-73- Config: cfg,
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/events/filesessions/fileuploader.go-74- }{
"file_path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/kube/proxy/forwarder.go",
"offset": 1,
"limit": 50
}1 /*
2 Copyright 2018-2020 Gravitational, Inc.
3
4 Licensed under the Apache License, Version 2.0 (the "License");
5 you may not use this file except in compliance with the License.
6 You may obtain a copy of the License at
7
8 http://www.apache.org/licenses/LICENSE-2.0
9
10 Unless required by applicable law or agreed to in writing, software
11 distributed under the License is distributed on an "AS IS" BASIS,
12 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 See the License for the specific language governing permissions and
14 limitations under the License.
15 */
16
17 package proxy
18
19 import (
20 "context"
21 "crypto/rand"
22 "crypto/tls"
23 "crypto/x509"
24 "encoding/pem"
25 "fmt"
26 mathrand "math/rand"
27 "net"
28 "net/http"
29 "path/filepath"
30 "strings"
31 "sync"
32 "time"
33
34 "github.com/gravitational/teleport"
35 "github.com/gravitational/teleport/lib/auth"
36 "github.com/gravitational/teleport/lib/defaults"
37 "github.com/gravitational/teleport/lib/events"
38 "github.com/gravitational/teleport/lib/events/filesessions"
39 "github.com/gravitational/teleport/lib/httplib"
40 kubeutils "github.com/gravitational/teleport/lib/kube/utils"
41 "github.com/gravitational/teleport/lib/labels"
42 "github.com/gravitational/teleport/lib/reversetunnel"
43 "github.com/gravitational/teleport/lib/services"
44 "github.com/gravitational/teleport/lib/session"
45 "github.com/gravitational/teleport/lib/srv"
46 "github.com/gravitational/teleport/lib/sshca"
47 "github.com/gravitational/teleport/lib/utils"
48
49 "github.com/gravitational/oxy/forward"
50 "github.com/gravitational/trace"{
"file_path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/kube/proxy/forwarder.go",
"offset": 50,
"limit": 20
}50 "github.com/gravitational/trace"
51 "github.com/gravitational/ttlmap"
52 "github.com/jonboulle/clockwork"
53 "github.com/julienschmidt/httprouter"
54 log "github.com/sirupsen/logrus"
55 "golang.org/x/crypto/ssh"
56 "k8s.io/apimachinery/pkg/util/httpstream"
57 "k8s.io/client-go/tools/remotecommand"
58 "k8s.io/client-go/transport/spdy"
59 utilexec "k8s.io/client-go/util/exec"
60 )
61
62 // ForwarderConfig specifies configuration for proxy forwarder
63 type ForwarderConfig struct {
64 // Tunnel is the teleport reverse tunnel server
65 Tunnel reversetunnel.Server
66 // ClusterName is a local cluster name
67 ClusterName string
68 // Keygen points to a key generator implementation
69 Keygen sshca.Authority{
"replace_all": false,
"file_path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/kube/proxy/forwarder.go",
"old_string": "\t\"context\"\n\t\"crypto/rand\"\n\t\"crypto/tls\"\n\t\"crypto/x509\"\n\t\"encoding/pem\"\n\t\"fmt\"\n\tmathrand \"math/rand\"\n\t\"net\"\n\t\"net/http\"\n\t\"path/filepath\"\n\t\"strings\"\n\t\"sync\"\n\t\"time\"",
"new_string": "\t\"context\"\n\t\"crypto/rand\"\n\t\"crypto/tls\"\n\t\"crypto/x509\"\n\t\"encoding/pem\"\n\t\"fmt\"\n\tmathrand \"math/rand\"\n\t\"net\"\n\t\"net/http\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"strings\"\n\t\"sync\"\n\t\"time\""
}The file /private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/kube/proxy/forwarder.go has been updated successfully. (file state is current in your context — no need to Read it back){
"replace_all": false,
"file_path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/kube/proxy/forwarder.go",
"old_string": "\tclusterSessions, err := ttlmap.New(defaults.ClientCacheSize)\n\tif err != nil {\n\t\treturn nil, trace.Wrap(err)\n\t}\n\tcloseCtx, close := context.WithCancel(cfg.Context)",
"new_string": "\tclusterSessions, err := ttlmap.New(defaults.ClientCacheSize)\n\tif err != nil {\n\t\treturn nil, trace.Wrap(err)\n\t}\n\tif err := os.MkdirAll(filepath.Join(\n\t\tcfg.DataDir, teleport.LogsDir, teleport.ComponentUpload,\n\t\tevents.StreamingLogsDir, defaults.Namespace,\n\t), teleport.SharedDirMode); err != nil {\n\t\treturn nil, trace.ConvertSystemError(err)\n\t}\n\tcloseCtx, close := context.WithCancel(cfg.Context)"
}The file /private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/kube/proxy/forwarder.go has been updated successfully. (file state is current in your context — no need to Read it back){
"file_path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/lib/kube/proxy/forwarder.go",
"offset": 178,
"limit": 30
}178 if err != nil {
179 return nil, trace.Wrap(err)
180 }
181
182 clusterSessions, err := ttlmap.New(defaults.ClientCacheSize)
183 if err != nil {
184 return nil, trace.Wrap(err)
185 }
186 if err := os.MkdirAll(filepath.Join(
187 cfg.DataDir, teleport.LogsDir, teleport.ComponentUpload,
188 events.StreamingLogsDir, defaults.Namespace,
189 ), teleport.SharedDirMode); err != nil {
190 return nil, trace.ConvertSystemError(err)
191 }
192 closeCtx, close := context.WithCancel(cfg.Context)
193 fwd := &Forwarder{
194 creds: creds,
195 log: log,
196 Router: *httprouter.New(),
197 ForwarderConfig: cfg,
198 clusterSessions: clusterSessions,
199 activeRequests: make(map[string]context.Context),
200 ctx: closeCtx,
201 close: close,
202 }
203
204 fwd.POST("/api/:ver/namespaces/:podNamespace/pods/:podName/exec", fwd.withAuth(fwd.exec))
205 fwd.GET("/api/:ver/namespaces/:podNamespace/pods/:podName/exec", fwd.withAuth(fwd.exec))
206
207 fwd.POST("/api/:ver/namespaces/:podNamespace/pods/:podName/attach", fwd.withAuth(fwd.exec)){
"pattern": "SharedDirMode",
"path": "/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037",
"output_mode": "content",
"context": 1,
"head_limit": 5
}/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/constants.go-550-const (
constants.go:551: // SharedDirMode is a mode for a directory shared with group
constants.go:552: SharedDirMode = 0750
/private/var/folders/04/qmf7f1qx7nj2_2mczkd4msth0000gn/T/bench-baseline-instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037/constants.go-553-
--
[Showing results with pagination = limit: 5]