Out of a sudden, one stream cannot be read
PlkMarudny opened this issue · comments
Which OpenObserve functionalities are the source of the bug?
streams
Is this a regression?
No
Description
Suddenly I cannot retrieve one of the streams
Please provide a link to a minimal reproduction of the bug
No response
Please provide the exception or error you saw
openobserve_1 | 2024-05-09T07:40:47.371396238+00:00 ERROR openobserve::service::search::datafusion::exec: [trace_id 2gDmJ6wpNKkuwLxmVlLURUrCUl7-wal-0] query sql execute failed, session: Session { id: "2gDmJ6wpNKkuwLxmVlLURUrCUl7-wal-0", storage_type: Wal, search_type: Normal, work_group: Some("global") }, sql: SELECT _timestamp,hostname,level,msg,pid,time,err_message,err_stack,err_type,err_code,err_errno,err_path,err_syscall,err_name,err_meta_body_error_reason,err_meta_body_error_root_cause,err_meta_body_error_type,err_meta_body_status,err_meta_headers_content_length,err_meta_headers_content_type,err_meta_meta_aborted,err_meta_meta_attempts,err_meta_meta_connection__openrequests,err_meta_meta_connection_deadcount,err_meta_meta_connection_id,err_meta_meta_connection_resurrecttimeout,err_meta_meta_connection_roles_data,err_meta_meta_connection_roles_ingest,err_meta_meta_connection_status,err_meta_meta_connection_url,err_meta_meta_name,err_meta_meta_request_id,err_meta_meta_request_params_body,err_meta_meta_request_params_headers_content_length,err_meta_meta_request_params_headers_content_type,err_meta_meta_request_params_headers_user_agent,err_meta_meta_request_params_method,err_meta_meta_request_params_path,err_meta_meta_request_params_querystring,err_meta_meta_request_params_timeout,err_meta_statuscode,module,lasterrorobject_n,lasterrorobject_updatedexisting,ok,value__id,value_agency,value_agency_content_1,value_audio_description_1,value_basename,value_cataloguing_status_1,value_content,value_copyright_1,value_cover_storage_key,value_created_at,value_description,value_description_1,value_duration,value_external_ids,value_grid_cover_storage_key,value_media_approved_by_1,value_media_filename,value_nl_updates_required_1,value_notes,value_notes_1,value_organization_id,value_origin_user_id,value_owner_user_id,value_recorded_at,value_restriction_summary_1,value_reuse_mam_warning_1,value_shared_organization_ids,value_skim_storage_key,value_source_date_1,value_summary,value_thumbs,value_title,value_title_1,value_type,value_updated_at,acknowledged,insertedcount,insertedids_0,insertedids_1,insertedids_2,insertedids_3,insertedids_4,insertedids_5,insertedids_6,insertedids_7,insertedids_8,insertedids_9,value_agency_1,0_status,0_value_acknowledged,0_value_matchedcount,0_value_modifiedcount,0_value_upsertedcount,100_status,100_value_acknowledged,100_value_matchedcount,100_value_modifiedcount,100_value_upsertedcount,101_status,101_value_acknowledged,101_value_matchedcount,101_value_modifiedcount,101_value_upsertedcount,102_status,102_value_acknowledged,102_value_matchedcount,102_value_modifiedcount,102_value_upsertedcount,103_status,103_value_acknowledged,103_value_matchedcount,103_value_modifiedcount,103_value_upsertedcount,104_status,104_value_acknowledged,104_value_matchedcount,104_value_modifiedcount,104_value_upsertedcount,105_status,105_value_acknowledged,105_value_matchedcount,105_value_modifiedcount,105_value_upsertedcount,106_status,106_value_acknowledged,106_value_matchedcount,106_value_modifiedcount,106_value_upsertedcount,107_status,107_value_acknowledged,107_value_matchedcount,107_value_modifiedcount,107_value_upsertedcount,108_status,108_value_acknowledged,108_value_matchedcount,108_value_modifiedcount,108_value_upsertedcount,109_status,109_value_acknowledged,109_value_matchedcount,109_value_modifiedcount,109_value_upsertedcount,10_status,10_value_acknowledged,10_value_matchedcount,10_value_modifiedcount,10_value_upsertedcount,110_status,110_value_acknowledged,110_value_matchedcount,110_value_modifiedcount,110_value_upsertedcount,111_status,111_value_acknowledged,111_value_matchedcount,111_value_modifiedcount,111_value_upsertedcount,112_status,112_value_acknowledged,112_value_matchedcount,112_value_modifiedcount,112_value_upsertedcount,113_status,113_value_acknowledged,113_value_matchedcount,113_value_modifiedcount,113_value_upsertedcount,114_status,114_value_acknowledged,114_value_matchedcount,114_value_modifiedcount,114_value_upsertedcount,115_status,115_value_acknowledged,115_value_matchedcount,115_value_modifiedcount,115_value_upsertedcount,116_status,116_value_acknowledged,116_value_matchedcount,116_value_modifiedcount,116_value_upsertedcount,117_status,117_value_acknowledged,117_value_matchedcount,117_value_modifiedcount,117_value_upsertedcount,118_status,118_value_acknowledged,118_value_matchedcount,118_value_modifiedcount,118_value_upsertedcount,119_status,119_value_acknowledged,119_value_matchedcount,119_value_modifiedcount,119_value_upsertedcount,11_status,11_value_acknowledged,11_value_matchedcount,11_value_modifiedcount,11_value_upsertedcount,120_status,120_value_acknowledged,120_value_matchedcount,120_value_modifiedcount,120_value_upsertedcount,121_status,121_value_acknowledged,121_value_matchedcount,121_value_modifiedcount,121_value_upsertedcount,122_status,122_value_acknowledged,122_value_matchedcount,122_value_modifiedcount,122_value_upsertedcount,123_status,123_value_acknowledged,123_value_matchedcount,123_value_modifiedcount,123_value_upsertedcount,124_status,124_value_acknowledged,124_value_matchedcount,124_value_modifiedcount,124_value_upsertedcount,125_status,125_value_acknowledged,125_value_matchedcount,125_value_modifiedcount,125_value_upsertedcount,126_status,126_value_acknowledged,126_value_matchedcount,126_value_modifiedcount,126_value_upsertedcount,127_status,127_value_acknowledged,127_value_matchedcount,127_value_modifiedcount,127_value_upsertedcount,128_status,128_value_acknowledged,128_value_matchedcount,128_value_modifiedcount,128_value_upsertedcount,129_status,129_value_acknowledged,129_value_matchedcount,129_value_modifiedcount,129_value_upsertedcount,12_status,12_value_acknowledged,12_value_matchedcount,12_value_modifiedcount,12_value_upsertedcount,130_status,130_value_acknowledged,130_value_matchedcount,130_value_modifiedcount,130_value_upsertedcount,131_status,131_value_acknowledged,131_value_matchedcount,131_value_modifiedcount,131_value_upsertedcount,132_status,132_value_acknowledged,132_value_matchedcount,132_value_modifiedcount,132_value_upsertedcount,133_status,133_value_acknowledged,133_value_matchedcount,133_value_modifiedcount,133_value_upsertedcount,134_status,134_value_acknowledged,134_value_matchedcount,134_value_modifiedcount,134_value_upsertedcount,135_status,135_value_acknowledged,135_value_matchedcount,135_value_modifiedcount,135_value_upsertedcount,136_status,136_value_acknowledged,136_value_matchedcount,136_value_modifiedcount,136_value_upsertedcount,137_status,137_value_acknowledged,137_value_matchedcount,137_value_modifiedcount,137_value_upsertedcount,138_status,138_value_acknowledged,138_value_matchedcount,138_value_modifiedcount,138_value_upsertedcount,139_status,139_value_acknowledged,139_value_matchedcount,139_value_modifiedcount,139_value_upsertedcount,13_status,13_value_acknowledged,13_value_matchedcount,13_value_modifiedcount,13_value_upsertedcount,140_status,140_value_acknowledged,140_value_matchedcount,140_value_modifiedcount,140_value_upsertedcount,141_status,141_value_acknowledged,141_value_matchedcount,141_value_modifiedcount,141_value_upsertedcount,142_status,142_value_acknowledged,142_value_matchedcount,142_value_modifiedcount,142_value_upsertedcount,143_status,143_value_acknowledged,143_value_matchedcount,143_value_modifiedcount,143_value_upsertedcount,144_status,144_value_acknowledged,144_value_matchedcount,144_value_modifiedcount,144_value_upsertedcount,145_status,145_value_acknowledged,145_value_matchedcount,145_value_modifiedcount,145_value_upsertedcount,146_status,146_value_acknowledged,146_value_matchedcount,146_value_modifiedcount,146_value_upsertedcount,147_status,147_value_acknowledged,147_value_matchedcount,147_value_modifiedcount,147_value_upsertedcount,148_status,148_value_acknowledged,148_value_matchedcount,148_value_modifiedcount,148_value_upsertedcount,149_status,149_value_acknowledged,149_value_matchedcount,149_value_modifiedcount,149_value_upsertedcount,14_status,14_value_acknowledged,14_value_matchedcount,14_value_modifiedcount,14_value_upsertedcount,150_status,150_value_acknowledged,150_value_matchedcount,150_value_modifiedcount,150_value_upsertedcount,151_status,151_value_acknowledged,151_value_matchedcount,151_value_modifiedcount,151_value_upsertedcount,152_status,152_value_acknowledged,152_value_matchedcount,152_value_modifiedcount,152_value_upsertedcount,153_status,153_value_acknowledged,153_value_matchedcount,153_value_modifiedcount,153_value_upsertedcount,154_status,154_value_acknowledged,154_value_matchedcount,154_value_modifiedcount,154_value_upsertedcount,155_status,155_value_acknowledged,155_value_matchedcount,155_value_modifiedcount,155_value_upsertedcount,156_status,156_value_acknowledged,156_value_matchedcount,156_value_modifiedcount,156_value_upsertedcount,157_status,157_value_acknowledged,157_value_matchedcount,157_value_modifiedcount,157_value_upsertedcount,158_status,158_value_acknowledged,158_value_matchedcount,158_value_modifiedcount,158_value_upsertedcount,159_status,159_value_acknowledged,159_value_matchedcount,159_value_modifiedcount,159_value_upsertedcount,15_status,15_value_acknowledged,15_value_matchedcount,15_value_modifiedcount,15_value_upsertedcount,160_status,160_value_acknowledged,160_value_matchedcount,160_value_modifiedcount,160_value_upsertedcount,161_status,161_value,162_reason_code,162_reason_result_deletedcount,162_reason_result_insertedcount,162_reason_result_insertedids_0,162_reason_result_insertedids_1,162_reason_result_matchedcount,162_reason_result_modifiedcount,162_reason_result_upsertedcount,162_reason_writeerrors,162_status,163_reason_code,163_reason_result_deletedcount,163_reason_result_insertedcount,163_reason_result_insertedids_0,163_reason_result_insertedids_1,163_reason_result_insertedids_10,163_reason_result_insertedids_11,163_reason_result_insertedids_12,163_reason_result_insertedids_13,163_reason_result_insertedids_2,163_reason_result_insertedids_3,163_reason_result_insertedids_4,163_reason_result_insertedids_5,163_reason_result_insertedids_6,163_reason_result_insertedids_7,163_reason_result_insertedids_8,163_reason_result_insertedids_9,163_reason_result_matchedcount,163_reason_result_modifiedcount,163_reason_result_upsertedcount,163_reason_writeerrors,163_status,164_reason_code,164_reason_index,164_reason_keypattern__id,164_reason_keyvalue__id,164_status,16_status,16_value_acknowledged,16_value_matchedcount,16_value_modifiedcount,16_value_upsertedcount,17_status,17_value_acknowledged,17_value_matchedcount,17_value_modifiedcount,17_value_upsertedcount,18_status,18_value_acknowledged,18_value_matchedcount,18_value_modifiedcount,18_value_upsertedcount,19_status,19_value_acknowledged,19_value_matchedcount,19_value_modifiedcount,19_value_upsertedcount,1_status,1_value_acknowledged,1_value_matchedcount,1_value_modifiedcount,1_value_upsertedcount,20_status,20_value_acknowledged,20_value_matchedcount FROM tbl WHERE (_timestamp >= 1715239547206000 AND _timestamp < 1715240447206000) ORDER BY _timestamp DESC LIMIT 250, err: Plan("Projections require unique expression names but the expression \"Int64(0) AS _status\" at position 93 and \"Int64(100) AS _status\" at position 98 have the same name. Consider aliasing (\"AS\") one of them.")
openobserve_1 | 2024-05-09T07:40:47.373578254+00:00 ERROR openobserve::service::search::grpc::wal: [trace_id 2gDmJ6wpNKkuwLxmVlLURUrCUl7] wal->parquet->search: datafusion execute error: Error during planning: Projections require unique expression names but the expression "Int64(0) AS _status" at position 93 and "Int64(100) AS _status" at position 98 have the same name. Consider aliasing ("AS") one of them.
openobserve_1 | 2024-05-09T07:40:47.373980923+00:00 ERROR openobserve::service::search::cluster: [trace_id 2gDmJ6wpNKkuwLxmVlLURUrCUl7] search->grpc: node: http://127.0.0.1:5081, search err: Status { code: Internal, message: "{\"code\":20008,\"inner\":\"Error during planning: Projections require unique expression names but the expression \\\"Int64(0) AS _status\\\" at position 93 and \\\"Int64(100) AS _status\\\" at position 98 have the same name. Consider aliasing (\\\"AS\\\") one of them.\",\"message\":\"Search SQL execute error\"}", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 09 May 2024 07:40:47 GMT", "content-length": "0"} }, source: None }
openobserve_1 | 2024-05-09T07:40:47.374135528+00:00 ERROR openobserve::handler::http::request::search: search error: ErrorCode(SearchSQLExecuteError("Error during planning: Projections require unique expression names but the expression \"Int64(0) AS _status\" at position 93 and \"Int64(100) AS _status\" at position 98 have the same name. Consider aliasing (\"AS\") one of them."))
openobserve_1 | 2024-05-09T07:40:47.374365960+00:00 INFO actix_web::middleware::logger: 10.20.64.28 "POST /api/ans/_search?type=logs HTTP/1.1" 500 244 "174" "http://localhost:5080/web/logs?stream_type=logs&stream=axf_json_worker&stream_value=axf_json_worker&period=15m&refresh=0&org_identifier=ans&quick_mode=true&show_histogram=true" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 OPR/106.0.0.0" 0.036689
Please provide the version you discovered this bug in (check about page for version information)
Version v0.10.5
Commit Hash 30d06f180da82120f75eb08b702eeb0e10438470
Build Date 2024-04-30T09:31:56Z
Docker image.
Anything else?
Have several streams, only one errors.
When I turn off "quick mode", at least I can display data, but no SQL query can be issued
Eventually, I have data that kills OpenObserve, just do not want to post it here as there is some confidential data there. Can supply if interested.
Got it:
error: 'Got 1137 columns for stream ans/logs/axf_json_worker, only 1000 columns accept. Data discarded. You can adjust ingestion columns limit by setting the environment variable ZO_COLS_PER_RECORD_LIMIT=<max_cloumns>'
guess there is a typo max_cloumns by the way.