Logging handler to send logs to your OpenSearch cluster with bulk SSL. Forked from https://github.com/logzio/logzio-python-handler
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

321 lines
11 KiB

9 years ago
9 years ago
4 years ago
9 years ago
3 years ago
7 years ago
7 years ago
7 years ago
8 years ago
8 years ago
8 years ago
3 years ago
  1. # The OpenSearch Python Handler
  2. ### Fork notice
  3. This project has been forked from [The Logz.io Python Handler](https://github.com/logzio/logzio-python-handler), version 4.0.1.
  4. After using the logz.io hosted and managed services to collect and visualize the logs of our Python appliances, we moved to
  5. a self-hosted on-premise OpenSearch cluster.
  6. To reduce the burden of adapting our applications, we modified `logzio-python-handler` to send the log stream directly to our OpenSearch
  7. cluster. This fork work as a drop-in replacement for `logzio-python-handler`, the name of classes is unaltered at the moment.
  8. Instead of the parameters `url` and `token` the LogzioHandler constructors now requires `host`, `port`, `username`, `password` of the
  9. OpenSearch cluster.
  10. Only basic HTTP auth is supported at the moment, and a behavios has been hard-coded to produce new indices for every month.
  11. <table><tr><th>
  12. ### Deprecation announcement
  13. Version 3.0.0 of this project ends support for Python 2.7, 3.3, and 3.4. We recommend migrating your projects to Python 3.5 or newer as soon as possible. We'll be happy to answer any questions you have in [a GitHub issue](https://github.com/logzio/logzio-python-handler/issues).
  14. Thanks! <br>
  15. The Logz.io Integrations team
  16. </th></tr></table>
  17. This is a Python handler that sends logs in bulk over HTTPS to Logz.io.
  18. The handler uses a subclass named LogzioSender (which can be used without this handler as well, to ship raw data).
  19. The LogzioSender class opens a new Thread, that consumes from the logs queue. Each iteration (its frequency of which can be configured by the logs_drain_timeout parameter), will try to consume the queue in its entirety.
  20. Logs will get divided into separate bulks, based on their size.
  21. LogzioSender will check if the main thread is alive. In case the main thread quits, it will try to consume the queue one last time, and then exit. So your program can hang for a few seconds, until the logs are drained.
  22. In case the logs failed to be sent to Logz.io after a couple of tries, they will be written to the local file system. You can later upload them to Logz.io using curl.
  23. ## Installation
  24. ```bash
  25. pip install logzio-python-handler
  26. ```
  27. ## Tested Python Versions
  28. Travis CI will build this handler and test against:
  29. - "3.5"
  30. - "3.6"
  31. - "3.7"
  32. - "3.8"
  33. - "3.9"
  34. We can't ensure compatibility to any other version, as we can't test it automatically.
  35. To run tests:
  36. ```bash
  37. $ pip install tox
  38. $ tox
  39. ...
  40. ```
  41. ## Python configuration
  42. #### Config File
  43. ```
  44. [handlers]
  45. keys=LogzioHandler
  46. [handler_LogzioHandler]
  47. class=logzio.handler.LogzioHandler
  48. formatter=logzioFormat
  49. args=('token', 'my_type')
  50. [formatters]
  51. keys=logzioFormat
  52. [loggers]
  53. keys=root
  54. [logger_root]
  55. handlers=LogzioHandler
  56. level=INFO
  57. [formatter_logzioFormat]
  58. format={"additional_field": "value"}
  59. ```
  60. *args=() arguments, by order*
  61. - Your logz.io token
  62. - Log type, for searching in logz.io (defaults to "python")
  63. - Time to sleep between draining attempts (defaults to "3")
  64. - Logz.io Listener address (defaults to "https://listener.logz.io:8071")
  65. - Debug flag. Set to True, will print debug messages to stdout. (defaults to "False")
  66. - Backup logs flag. Set to False, will disable the local backup of logs in case of failure. (defaults to "True")
  67. - Network timeout, in seconds, int or float, for sending the logs to logz.io. (defaults to 10)
  68. - Retries number (retry_no, defaults to 4).
  69. - Retry timeout (retry_timeout) in seconds (defaults to 2).
  70. Please note, that you have to configure those parameters by this exact order.
  71. i.e. you cannot set Debug to true, without configuring all of the previous parameters as well.
  72. #### Dict Config
  73. ```
  74. LOGGING = {
  75. 'version': 1,
  76. 'disable_existing_loggers': False,
  77. 'formatters': {
  78. 'logzioFormat': {
  79. 'format': '{"additional_field": "value"}',
  80. 'validate': False
  81. }
  82. },
  83. 'handlers': {
  84. 'logzio': {
  85. 'class': 'logzio.handler.LogzioHandler',
  86. 'level': 'INFO',
  87. 'formatter': 'logzioFormat',
  88. 'token': '<<LOGZIO-TOKEN>>',
  89. 'logzio_type': 'python-handler',
  90. 'logs_drain_timeout': 5,
  91. 'url': 'https://<<LOGZIO-URL>>:8071',
  92. 'retries_no': 4,
  93. 'retry_timeout': 2,
  94. }
  95. },
  96. 'loggers': {
  97. '': {
  98. 'level': 'DEBUG',
  99. 'handlers': ['logzio'],
  100. 'propagate': True
  101. }
  102. }
  103. }
  104. ```
  105. Replace:
  106. * <<LOGZIO-TOKEN>> - your logz.io account token.
  107. * <<LOGZIO-URL>> - logz.io url, as described [here](https://docs.logz.io/user-guide/accounts/account-region.html#regions-and-urls).
  108. #### Serverless platforms
  109. If you're using a serverless function, you'll need to import and add the LogzioFlusher annotation before your sender function. To do this, in the code sample below, uncomment the `import` statement and the `@LogzioFlusher(logger)` annotation line.
  110. **Note:** For the LogzioFlusher to work properly, you'll need to make sure that the Logz.io. handler is added to the root logger. See the configuration above for an example.
  111. #### Code Example
  112. ```python
  113. import logging
  114. import logging.config
  115. # If you're using a serverless function, uncomment.
  116. # from logzio.flusher import LogzioFlusher
  117. # Say I have saved my dictionary configuration in a variable named 'LOGGING' - see 'Dict Config' sample section
  118. logging.config.dictConfig(LOGGING)
  119. logger = logging.getLogger('superAwesomeLogzioLogger')
  120. # If you're using a serverless function, uncomment.
  121. # @LogzioFlusher(logger)
  122. def my_func():
  123. logger.info('Test log')
  124. logger.warn('Warning')
  125. try:
  126. 1/0
  127. except:
  128. logger.exception("Supporting exceptions too!")
  129. ```
  130. #### Extra Fields
  131. In case you need to dynamic metadata to your logger, other then the constant metadata from the formatter, you can use the "extra" parameter.
  132. All key values in the dictionary passed in "extra" will be presented in Logz.io as new fields in the log you are sending.
  133. Please note, that you cannot override default fields by the python logger (i.e. lineno, thread, etc..)
  134. For example:
  135. ```
  136. logger.info('Warning', extra={'extra_key':'extra_value'})
  137. ```
  138. ## Django configuration
  139. ```
  140. LOGGING = {
  141. 'version': 1,
  142. 'disable_existing_loggers': False,
  143. 'formatters': {
  144. 'verbose': {
  145. 'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
  146. },
  147. 'logzioFormat': {
  148. 'format': '{"additional_field": "value"}'
  149. }
  150. },
  151. 'handlers': {
  152. 'console': {
  153. 'class': 'logging.StreamHandler',
  154. 'level': 'DEBUG',
  155. 'formatter': 'verbose'
  156. },
  157. 'logzio': {
  158. 'class': 'logzio.handler.LogzioHandler',
  159. 'level': 'INFO',
  160. 'formatter': 'logzioFormat',
  161. 'token': 'token',
  162. 'logzio_type': "django",
  163. 'logs_drain_timeout': 5,
  164. 'url': 'https://listener.logz.io:8071',
  165. 'debug': True,
  166. 'network_timeout': 10,
  167. },
  168. },
  169. 'loggers': {
  170. 'django': {
  171. 'handlers': ['console', ],
  172. 'level': os.getenv('DJANGO_LOG_LEVEL', 'INFO')
  173. },
  174. 'appname': {
  175. 'handlers': ['console', 'logzio'],
  176. 'level': 'INFO'
  177. }
  178. }
  179. }
  180. ```
  181. *Change*
  182. - token - Your logzio token
  183. - url - Logz.io Listener address
  184. - logs_drain_count - Number of logs to keep in buffer before draining
  185. - logs_drain_timeout - Time to wait before draining, regardless of the previouse setting
  186. - logzio_type - Log type, for searching in logz.io (defaults to "python"), it cannot contain a space.
  187. - appname - Your django app
  188. ## Trace context
  189. If you're sending traces with OpenTelemetry instrumentation (auto or manual), you can correlate your logs with the trace context.
  190. That way, your logs will have traces data in it, such as service name, span id and trace id.
  191. To enable this feature, set the `add_context` param in your handler configuration to `True`, like in this example:
  192. ```python
  193. LOGGING = {
  194. 'version': 1,
  195. 'disable_existing_loggers': False,
  196. 'formatters': {
  197. 'logzioFormat': {
  198. 'format': '{"additional_field": "value"}',
  199. 'validate': False
  200. }
  201. },
  202. 'handlers': {
  203. 'logzio': {
  204. 'class': 'logzio.handler.LogzioHandler',
  205. 'level': 'INFO',
  206. 'formatter': 'logzioFormat',
  207. 'token': '<<LOGZIO-TOKEN>>',
  208. 'logzio_type': 'python-handler',
  209. 'logs_drain_timeout': 5,
  210. 'url': 'https://<<LOGZIO-URL>>:8071',
  211. 'retries_no': 4,
  212. 'retry_timeout': 2,
  213. 'add_context': True
  214. }
  215. },
  216. 'loggers': {
  217. '': {
  218. 'level': 'DEBUG',
  219. 'handlers': ['logzio'],
  220. 'propagate': True
  221. }
  222. }
  223. }
  224. ```
  225. Please note that if you are using `python 3.8`, it is preferred to use the `logging.config.dictConfig` method, as mentioned in [python's documentation](https://docs.python.org/3/library/logging.config.html#configuration-file-format).
  226. ## Release Notes
  227. - 4.0.1
  228. - Updated `protobuf>=3.20.2`.
  229. - Added dependency `setuptools>=65.5.1`
  230. - 4.0.0
  231. - Add ability to automatically attach trace context to the logs.
  232. - 3.1.1
  233. - Bug fixes (issue #68, exception message formatting)
  234. - Added CI: Tests and Auto release
  235. <details>
  236. <summary markdown="span"> Expand to check old versions </summary>
  237. - 3.1.0
  238. - Bug fixes
  239. - Retry number and timeout is now configurable
  240. - 3.0.0
  241. - Deprecated `python2.7` & `python3.4`
  242. - Changed log levels on `_flush_queue()` method (@hilsenrat)
  243. - 2.0.15
  244. - Added flusher decorator for serverless platforms(@mcmasty)
  245. - Add support for `python3.7` and `python3.8`
  246. - 2.0.13
  247. - Add support for `pypy` and `pypy3`(@rudaporto-olx)
  248. - Add timeout for requests.post() (@oseemann)
  249. - 2.0.12 - Support disable logs local backup
  250. - 2.0.11 - Completely isolate exception from the message
  251. - 2.0.10 - Not ignoring formatting on exceptions
  252. - 2.0.9 - Support extra fields on exceptions too (Thanks @asafc64!)
  253. - 2.0.8 - Various PEP8, testings and logging changes (Thanks @nir0s!)
  254. - 2.0.7 - Make sure sending thread is alive after fork (Thanks @jo-tham!)
  255. - 2.0.6 - Add "flush()" method to manually drain the queue (Thanks @orenmazor!)
  256. - 2.0.5 - Support for extra fields
  257. - 2.0.4 - Publish package as source along wheel, and supprt python3 packagin (Thanks @cchristous!)
  258. - 2.0.3 - Fix bug that consumed more logs while draining than Logz.io's bulk limit
  259. - 2.0.2 - Support for formatted messages (Thanks @johnraz!)
  260. - 2.0.1 - Added __all__ to __init__.py, so support * imports
  261. - 2.0.0 - Production, stable release.
  262. - *BREAKING* - Configuration option logs_drain_count was removed, and the order of the parameters has changed for better simplicity. Please review the parameters section above.
  263. - Introducing the LogzioSender class, which is generic and can be used without the handler wrap to ship raw data to Logz.io. Just create a new instance of the class, and use the append() method.
  264. - Simplifications and Robustness
  265. - Full testing framework
  266. - 1.X - Beta versions
  267. </details>